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ABSTRACT 



The value systens \>\ vhich the use^r's of purc&ased 
information services select vhich one to use or bi^. are found to 
differ sharply aaong different, oser populations, but the variable 
found to control the« is a function of use, not of the tiser as an 
individual. Selection variables are suasarixed in a Mtrix of user 
values a^nd interaction effects* The reseucher prefers -original 
docuaents, ajid ter^ds to choose sub Jectivedye The oipierating aanager 
■orQ often stopd with abstracts^ and emphasizes guantifiable factors* 
The planning Manager wants a -digest* pf ideas, not references, and 
tends to judge a service first as to i^s. reliability. These three 
functions attach entirely difteremt connotations to the saae words 
for key paraseterse They can be linked to t^he structure of language: 
operations is the present participle (doing), research concerns the 
verbal^adiecti ve (state of the art), and planning is in the 
sublunctive node (could or shou].d) • tpparetit conflicts, whlc^ say be 
resolved by disensional analysis of difference in values perceived, 
are illustrated by case histories,' interviews, &nd guestionnaire 
results. (Author) ' • » . 
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FOREWORD 



This is a firrst-year report of research, on improverwents in the use of scientific and 
technical information (STI) supported by the Division of Science Information (DSI), 
National JScience Foundatvon. It provides a systematic investigation of criteria 
applied in "decision-making by a* wide variety of individuals who .are users or 
managers of purchased information services in the commercial sector of the 
infor-mation industry* DSI is interested in criteria for the comparative evaluation 
o{ inforcnation services, including variables thgtt influence cost and thos^ that^do 
not* We are supporting, research on mHhods to raeasure the value of these 
services, ^s part of our overall program to improve^ the ^efficiently and 
effectiveness of the utilization of STK " 

• ., ^ 

Your suggestions ^nd comments on this repprt will be appreciated. 
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Director 

^Division of Science Information 
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, ' user' VALUES IN THE SELECTION • " , , 

OF INTORMATION SERVICES 
^ ) - - • ' . 

. \ ■ - HigHLIGHTS ' : • 

Which infbrmation service to use, and-why? Us^ts* preferences ' 
among services and their variations from time to time are analyzed in this 
study for the User .Jtequiremehts Section of NSF, The controlling variable.- 
was, founcf to be the function for which processed information is desired: ' 
for research jjhow to do things), for planning (what^ to do), or for Informa/ 
tiori operations (do iiii^ things with the infonoation' itself )!• User popula- 
tions which differ in^heir values for. pulrcha^ed services< can be grouped ' 
according to this use functiot^. * The services Considered were categorized 
as* access tools for direct use, or paid searching, or information analysis 
and evaluanbii. 

*' • • » ' ' 

The principal finding of this study is that ^he value system which 
a given user applies in seiecting'^an information ^'service varies from time 
to time, depending upon use as an event- rather than the user as an 
individual. Ther^ is k subtle but distinct difference from the initial 
proposal for this study, which was to identify groups of users who agree and 
disagree and t^o explore the characteristic patterns of use by. which they 
might be distinguished. The identifying variable is useful to a vast 
variety^, of users: a^iven person may be acting as a scientist today, a 
se'archer tomorrow, and a planner the next afternoon, and the service he 
prefers will depend uport his function at the moment. This functional f ela- • 
tionshl^p may explain why previous user studies tend to gloss over the 
behavior of the^ tndlvldualr-as being an erratic and ui5r1adlctable uAknown, 

* * * 

Each group of users has its own set of priorities a^ special 
connotations for variables in .the selectlorf process. WoAers* ^lyolved in 
^thQ research/development fuiictlon (scientist, ^ftgliie^ or patent attorney) 
place high value on subjective factors of sprVlce quality or convenience. 
Those involved in information operations (searching or retrieval) are ;iipre 
coticei^ned with the quantifiable- faptors of time,, cost, and facilities' \ 
available^ Planning (management, staff, or individujit) is morB judgmental, 
and aeeks aif Individual whose purchased service can be trusted before even 
considering factots'of cost or convenience. Managers^'or users of informa- 
tion services whose function the moment is research or operations or 
planning may attach quite different connotations to th6^ same key words/ 
such as useful information, pertinence vs. relevance, depth vs; breadth, 
timeliness^ time, and costs. The result I9 a conflict in valuer, systems 
which is .often unrecognlzfed. • 

This b^slc finding and its impJApattons were^ developed by direct ; 
^^teinrlewB with some 60 individuals., '^ technique applied was a dimen- 
sional^ analysis Qf the factors in the sW^ctlon process which each. user 
cotjbiders most important. Selection factbrs'can be grouped in three 
dlMensions as quantifiable, tjualitative, and juflgmental:. these are all 
recognized as important by all user ^oups,' but with sharp differences 
in the telatlve Values th^y per<;^lve. The emphasis on the individual was a 
direct opposite of whAt some people prefer when they use the "Delphi 
approach" and mask all personal dlffel^ences in anonymity, to arrive at a , 
statistical consensus. L^sts of discriminant factors were first prepared in 



lntervle%« witfh sppcidljsta who spend oxer- 30Z of their, time" In information 
research, ^and expanded to include users who are scientists,, engineers, 
attorrg^s, and planning staff. Each *f these populations ^included four 'to 
six individuals in th^ basic^roup of 30 Interviews in-house. ' Preliminary 
contli/sions were augmented and confirmed by interviews with an eq^ual number 
oy users and managers of information services in other companies. These 
drew special attention to the experiential and intuitive components in manage- 
mervt judgment,' as| distinct from the quant if ialJle/rational and qualitative/ ' . 
8ubject4.ve compp«ents in conventional evaluations of cost effectiveness and 
cost/benefits. . % ' 

0 

' A Matrix diagram is proposed to' sUmniarize the. diverse effects of 
many factors afe interactions in the user's selection process. The major 
variables are entered in matrix rows: selection factors, user populations, 
and types of informatien service. Sub-«elembnt d^ensions ^Are expanded in 
separate charts. The columns 'in the matrix represent how these variable^ 
are defined by 'function, how they perceived by the*user, .and how they 
interact. One interaction effect is the difference. Ih^ priorities of dif- 
ferent user groups. Other interacticms involve the environment of the ^use- 
&fent: user concerns', management cdfcerns, and maturity of the project. 

« A questionnaire was de'VIB.oped at the end of the study to ^xamine 
' the acceptability, of the parameters ??elected and the definitions proposed 
fo^ them^ This was addressed tb &9, individuals who consider themselves as 
users of imformation services, out of a random %atnple of 240 professional 
employees at Exxon. . A preliminary analysis of the returns shows significant 
differences between the statistics for research scientists and for engineers. 
Only a third of the engineers had used the published literature to. search 
for information dCrine tffe year 1975-76, vs. nearly half of the scientists. 
Scientists tend to value the the published information 'more', ^nd they gave ^ 
a much higher response to Jfcf iSitt|Stionnaire. • V^ey used or specified more ^ 
services by title during ni^^iiflr. and r1|juested many more searches. Perceived 
values were strongly posit^e^r^r scientists negati've- for engineers, in 
preferring original document^ yjsrsus abatracts^ requesting the,tise df a 
specijjic service, of asking a kftown individual to collect the information^ 
Thfis preference for a known pW90h was even sTCfonger for sei^rches ordered ^by ' 
patent attorneys, who cited it a^ their priaayy basis for selecting a service. » 
All user's agreed that they want to^et key reference^ promptly before a fur- 
tne'r search, and there was a strong vote against ' expect iifg t,he customer who 
•^ets the information 'to know how to' use a* Computerized system. It must be 
recognized that these vreplie^ were in an environment where information* services 
are ^ailor-igade- to company interests', and available on request. 

'0 Case histories ,de\7felpped dpring th^ X^^^Y ft^ve shown that this 
approach can defuse many areas of pfotential conflict. The junior staff 
employee need not feel that a management change In his recommendations 
implies a criticism or lack of confidence; a n^w business, which is smalJL 
still needs bqth a broa<i view of information for corporate planning and 
specific details 'fpr ft&D; the information specialist in a small 
library may ^be valued ojost for non-computerizable skillsfin upgrading the 
input. to the system,^ hot as a paid searcher. It was possible in each of 
these cases to detect points of view which could be easily ifecognized by 
•parties who.had been in disagreement, as different but not opposites. 
Accepting ^these as a valid' basis for joint action avoided the destructive 
effebts of^a forced choice between them* : 



• ^ " gC^qi^IVE, SUMMARY ^ ' - 

Improvements ;ln the dissemination ^rf ,use of jscl^At^f Ic ud • 
technical Information have created* a grovth Industry of competing Informa- 
tion services whose sunflval depends upon uBers and customers. Users 
differ,' and the value systems by which they decide ^whlch service to select 
at a given time are a complex function of Identlf^ble variables and , 
relatlojashlps^ The present study has been conducted for the User 
Requlreiaents section of the liSF* tto explore twd questions: who wants to 
use what services, and ho^does the user decide?* . • - 

I ' . ... , ^ 

^ Different gronps of users are found fo differ sharply in the 
priorities they assign to major variables In the dfelectlon process. A 
primary' go^ of this study ^s to help each user to Identify \L±a or her 
own patterns of' use. RelatJli* goals are to explore weaknesses In the 
user's methods of selection 'and -ij^re to look for help, to recognize 
others whose points of view are*slallar or different, and to benefit 
more, readily from shared experience^as a rfesult. This is an analysis 
of controlling variables from the viewpoint of the user, a^ distinct 
from that of the designers or vendors of se^i,ces who have contributed 
most of the literature on the subject. Field of coverage is an important 
variable but it is excluded from this study^by definition, since if 
there is only one service In a gi\Ken field there is no competition. ' 

the selettldn between services is seeji as a dynamic proceSs, 
in wl>lch each user can assign his (her) own values to the factors Involved. 
These value judgments can, change for the same person for each use event, 
depending on the environment of use, the typjs of service, and the 
function for which the information' is desired* It is necessary for this 
discussion to give ^lear and arbltr^i;^ def lnitio*8 to a set of basic 
"terms; ^ • * ' 

SIX - the whole gamut of recorded seientlfic and technical 
information, and associated access tools. 

• Purchased Information servic^e - a comov^rclal enterprise 
' providing its uffers and customers wi^h processed ^ 
information, which saves them from haVing to get it 
from original sources. 

User - the one who selects which Wormation service to 
•buy or use and works directly with the service selected > 
so choice la based on personal experience. 



National Science Foundation Division of Science information (formerly 
NSF/OSIS) Contract C-1027 with Exxon -Research and Engineering -Company, 
on "Improvements in the Dissemination and Use^ of Scientific and 
Technical Information", funded June 30, 1975; Homer J. Ha^l,, 
Principal Investigator^ (retlrSd, December 1976) 
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Customer - the one who pays for the Information^ supplied, 

directly or indirectly, but may not have an^- part in'' 
, the use of the service or in the selection' process. 



\ 

\ 



* \ User population - an identifiable group of users who ) have the ' * 

\ • same basic needs and values fpr i^ormatiou services^ "^he 

\ difference lies in what function the user is exercising at - 

a given time: Ihe same individual may be functioaing as a , 
scientist today, a searcher Jtoiaorrow, and a planner the * • , - • 

next af tf^oonl / « / . ' ^ ' . 

User populations and some oT their characteristic simUaritie/ - ' ' . 
\ and differences have been examined in this s,tudy by interviews with 60 

known 'users or managers of ^information services, half of them within ER&E- * 
• and half outside. This included a strong pijpgram of field discussions and ♦ 
' conferences during formative st^es, to test and refine the j^rellminarj 
results. Conclusions drawn on this basis were then checked by questiort- *' 
naire, based on^a 2407name random sample , of 'profes9iorial employees' at .ER&E 
who are potential users of purchased services. - . 

A Matrix diagram is shown whix:h summarizes the differences observed ' , 
in user values and their interaction effects.' The major variables for analysis 
are presented in the matrix rows, ^ selecUcfti factors^ user populations, and ^ 
types of information service.^ The matrix colics represent how these variables 
r are defined, how tlie/ are percelvec^ by the use\ and how they interact. 

There are many discriminant factors which can be considered 
^significant in the selection process. -Initial interviews suggested that 
/ these can- all be categorized as quantifiable, ^qualitative, or judgmental. ' 
Variables in each of these dimensions are so linked that they tend to move 
together in any change or comparison. For example, ds^ta on cost, time, 
physical facilities, or personnel can be considered "as one such group, in 
the dimension of^ quantifiable factors. That is to say, any circumstance 
whi^rh has a major effect on cost;,,, time, facilities*-, or staff req\£.rements ' ^ 

is likely to have some effect on ot^ier variables* in the same group. This 
concept of dimensions in evaluation is taken frpm earlier studies on <he 
value of research projects, 

• / ' 

Ji second dimension which ip well recognized includes the whole' 
gamut of qualitative factors that cannot be measured exactly but which * 
can be handled by the technique of subjective ranking. The exact' naAes ' 
given to such information service concepts as conveniepce, flexibility 
or responsiveness ai^e hard €o define, but they tend to be interrelated 
and move Xogether when any one of them is significantly changed. 

The decision process also involvfes factors o£ judgment which 
can have a yes/no answer. Judgment is a function of management, and the ' 
manager may be the same perspn as the user or someofie else. The Judgmental 
dimensiop^is hard to argue with, based on neither numerical facts Aor 
subjective rankings alone but a "comnion sense" blend of ^hese with intuition, 
based on experience. Reputation, suitability or future prospects often 
involve a major contribution from the judgment of other managers. 9lutside * ' , 
of management these factors are often deferred to others and then taken^ ' • 
for granted . ' . . 

^ / ' 
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MATRIX OF USER VALUES' AND TNTERACTjlQNS 



J 



Parameters 



Variables* 



De fine d as 



Select ion-^actcfr. Dlaensioh^ 



^Quaptitiable 
Qualitative . 
Judgmental 



Budget/coats 
Coi^venlence 
Reputation . 



User Populations 



Sclent Is^ » engineer , ' 

attorney 
Searcher, 

Information analyst 
Manager ^ Planning function 



R/D functltfitx 
Info operations 



Purchased Information Services S 



Access tools 

Searching 

Analysis 



^elf-servlc^ 
. l*re-def Ined^ questions 
Optimized during search 



Fercelved -^Values 



Rational' 

Subjective 

Intuitive 



Wants^ 
Original.s 
/ 

Abstracts 
abs + or±g 
. EvaluatJLoi*^ 



Impersonal ^ 
' Intermediary 
• Highly indlv^ual 



*^ Interaction Effec'ts 




Users' 'Priorities 




Environment of use 
U'ser-worker concerns 
Mana^emei^t concerai^ 
Maturity of iptoject 




Variables as iisted are typical, not all- inclSeive, 



' The- specif ic factfpts- ilBte^ 4ret>typical of many others. The 
, Ifljportarit point, is tl)4t- dimensions *ln* the- selection pwcess* can^ ; 
recognized, jSgard^legs of . the ^exact nimes^a^^^ociatedtwitK th^m, an,d that 
theyipan be used*to characterize jaTt^. c^f/tjie' s.ele/:tion process. The 

* dimensional' apprb.ach has ^Ke advantkg^ thafc' the categories named are • 
compl^mentaxy, nqt opposite^^^nd^^h:ei*«|ora^*les.9 likely ' to leave holes 
in the matrix., * 4 \^ - • ^ 

The literature di^ the relative ijjaportance of^ujser-selcction 
factors is^iased at this, point, .since it depends largely on who wrote - / 
what. Art icl^es. which en\pliasi^ the 'importance of cost tejid* fb' be written 
by 'vendors, including in-hoilse managers, who have" a/seri^c^ to sell. , 
'Both are continually asking in the '-literature tor 'c«st "data 'from- others * 
to compare With their 6wn.' Other autfiors examine costVbenef i.t telatien-. / 
ships that, inclujde subjective , factors. This viewpoint's characteristic 
-pf the social scientist." Th^ Judgmental approach, »is commonly recognized 
in jouflrnals of indu.strial mana^iem^iit but *it can* be anathema, to the '"purely 
rational person" who prefers budget numbiers Wd considers' even coat/benefit. 

* ratios as a compromise. These strong differences* in priorities aiW points 
-of vi^^ tend to confirm the validity of i^ing the dimensions chosen, as 
^separate factdfs in the decision process." TH^ is- not to say tHat -any one 
•user, of services v^jj.1 base^ his choice, entire^ on one set of values:-* all 
of them are-important* . What it does say is ^at whii€ the ^atoe v^ri^bles 
apply^ to all users, they ar^dif f erently perceived. ' 

/ * ' > ' - / 

♦ , ' * User Populations ^ 

The central featute of the majtrix comprises thr^e groups of .u6er 
populations which can be distinguished by function and how they* use the* 
information supplied. This -distinction is shpwn by whether the.infortha- * 
tion they most want to have' "h^nds-on" is or^Cginal documentSji* selec^d 
abstracts and- reference^yOf evaluated overyiews.^ First is the research ^ 
functioi^ which includes scientist's, engineer^, and^atent attorneys, profes^ 
sional men whose information, need is, f or -^Igtrfal dltum^nts, but/tor - 
different reasons* The scientist wants .to sfAdy and .com^r^ different 
set?B ,of briginal data, or the i^ethods by which they were obta:ined. The 
engineer wants a single set of reliable data add the 9ondltia^s held 
constant to obtain them, but jiot all the data there ar.e. TKe attorney 
drafting a patenter a new agreement^ needs the c«mplete*and exact wording 
of selected originals, but is less cfoncerfted about cfata.?^" ^ 

• ^Ihe information operation of searching (or tlie individual who ; 
is acting* *^s a paid searcher a^ijd not as a scientist or as a .manager) is 
frequently satisfied to stop with ai* abstract or reference by title only. . 
-This identifies useful source documents and K^ere to find them, but'does' 
not require their physical production. ' . . " 

Planning as b function of management pay be /delegated more jor less 
completely to planning Staffer fo othfer^s* Planni?\g preffers an overview or 
information digest, for the selection of target areas. Typically pressed for 
time, the planner "^wants immediate access to all available viewpoints and 
suggestions on how to evaluate them/ to decide which ones , to pur^e.. He is ' 
looking for ideas and Is less concerned at^ the moment with otiginals, or 
-where to , find them. ' . • * ' " ' ■ ' 



I 
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JPrloritles ^ 



Th6 simplistic statement for any uset who has a choice Is that 
he wants "^the be^t serylce he can afford, for-^a given use/' Thlp dif- 
ference In function , Spends upon the specific uae event, rather thjan 
emphasising the variable need3 of axglven Individual. Interviews with 
users, in different populations suggested comp;ieteIy different priorities 
in thdse t^lationshipsV. 



- The research scientist/engineer clfooses f irst ^o 
service he finds most convenient to Use. He a^epfT 
the judgment of a manager as tp what systems to consl^ 
but he pays little attention to the quantifiable factors 
of cost, tike, or facilities as long as the systems avail-, 

• able give him th^ ^answers .lie needs. 

- The planning staff or manager makes arfiVst choice -on . 
whether the serylce In ^ttestfpn is rellablfe ^nd "Suits . ^ 
our way of doing things/" He has to* consider budget! 
on aj^l^he quantifiable factprs,' but m^y include . 
qualitative factors only after. h^ has ruled Out services 
which he finds xinreliable'^jor too expensive. 

- The. literature on information operations emhaslzes cost, 
tlme^, and ail the quantifiable factors a vi^ndor must 
control to keep Md customers. * Quaflitative. factoss mky ^ 
be recognized in costyben^f it studies, 'but somewhat 

*. grudgingly because they cannot be accurately measuted. 
^ * • Judgment as a, separate dlmdnsion is JlPt so clea^ix 

recognized in *th^e J^lterature on infijfrmktion services ks 
It is in didcu?sions on 'business m^^agement. \ 

it must be emphasized tliat these "priorities" are all ^Idtive 
values, not absolute. Cost data, fdr examp^e^ .appear to be le,ss signi- 
ficant to the scientist than convenience ot^ rel^aliility: this is not to 
say that cost data^ can be ignorfed, but qnly that the scientist is more 
likely than the vendor or jj^^ager to put other factor^ ahead of cost. 

Piirchdsei services from whfch ajiy user may select fall into 
three major categories: . . ^ 

' access tools for direct use, at all levels pf complexity. 
^ These^tnciude indexes, abstracting services, and computer' 

data bases l4 routine use. 

p \] 7 a pulchafted service that does searching for known facts, 
.where^tlie question is defined before the search begins; 

- a service for Information analysis, where the user pays 
- " # someone* else both to find anH to analyze the Information 
» available.. ' ^ ^ 
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The same iniormatio^ services, ev*^n.thfe most sophisticated . , 
ones^i may be uped for self-h§lp, for paid sear<?hing, or for infprmation 
analysis: the difference lies i^'the userj an^ how the' information * 4 
'tools aiid^services ^re applied. In general,, the information analyst 
is applyj^'a higher, lev/el of skills than in searching alone, becaus^ 
"the^nalyst njust^ under st/ind the technical language of the literature and 
th^^stonjer |s needs, ^ well' as the inner workings of the searching 
procedures iised.;- , 

. ^ Environment of the Use-Event 
— " " 

* Interactions with the environment of user concerns ^ maijagement 
concerns, and specific project are an active part of the selection process, 
where the three types of service defined' differ signif icantlyTn their effects* 
The routine use of in index or abstracting service is perceived as impersonal 
and for this the user/worker will acfcept "any competent assistancB« The user 
who is buyipg a literature search is "more concerned, ta be sure that the' 
intermediary employed understands the po»ints agreed upon in negotiating the 
questjion/ The user who has asked for infbrmat4pn analysis paying someone 
to think .for him. This is a highly/individual matter involving the user's 
self-esteem, and he wants to know exactly who the analyst is. 

>^ 

Personal impacts are high in a situation where th^ user feels 
that he is applying his highest ^ills. This interacts with user function: 
research is concerned with, findljng ^aps or inconsistencies; the searcher 
is concenfed with cornijjetin^ thi reference file; the planner *may consider 
one of' his Q^ssemtia^l taaks aVl/rowsing in selected .sources toffind useful 
results from another field. An whatever area each user considers ci;itical V 
he is likely to insist on thfe services of a person he feels he can trust, . 
or he may reject any surf^-hfelp as undesirable interference and do this part 
•^of the work himsel/ 

Other/ impacts of the environment include both limitations ^on 
the user and positive effects. A guiding principle here is Mooer's law, 
which. says .that the user of information tends to avoid any action which 
gives him pain. For example, freedom of choice can be sharply restricted,, 
by limited authority for cost approvals,. Within the user's limits of ^ 
approval, the "free vs. fee" argument may impose a forced cost deci'sion 
on users whosfe value system would always place convenience or quality first. 
Their skills in the budget/cost area may be so poor that a forced choice 
is arbitrary oiv capricious, ignoring cost/benefit trade-offs which they 
normally defer to Someone else. Conversely, utilization may be improved 
by setting up for the. user clearly defined blacks of services for which 
approval for him is free, or easy to get, 01^ more difficult jjor special 
• Situations.' , ^ 

ll Another user concern is the capability of the individual to 
use the information supplied. This is not the, same thing as knowing hoW 
to use the service, nor does it mean that he is actually going to use it. 
\ It helps to avoid overkill or .answering the wrong question; Information 
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can be tested out . In many ways,, say by computer modelling or by discussion* 
The point ^Is that the selection and use of Information Is easl^t arfd most 

* effective irt areas wlth^ the. user's experience. Personal Interest* and 
self-esteem may lead the user to Invent reas*ons £ot drying the newest an4 
most so|>hlstlcated. techniques^ as part of the learning process* Values In 
such a case accrue* In actual ^se, riot potential use, as witnessed by the 
experlence^Df firms who overbought on computers as a status- symbol*. , 

0 \ , ^ 

In an Information operation where performance standards are main- 
tained, a'ccuracy and completeness may be taken for granted. This Is not 
always true, and where It Is not these paraipeters (Quickly beccmie vitals 
Vulnerability to poor data or negative information varies with the specific 
use: it is i^gh for a quick answer taken at face value. Even unreliable^ 
results which are ptoperly reported may be valuable in research for .detect- 
ing a result which seems out of line. This can be ^ nuisance in searching 
and even more so in- planning, • unless the ^certainty Itself is important* , 
A definite lack of "concern for the scientific Quality of Information applies 
to certain- operations of' the planner surveying a new field, who wants the 
widest variety o^ points, of view available. He Is^tiot necessarily concerned 
with whether a pew viewpoint corner ^om -a scientist or a tabloid columnist. 
The tabloid bias .may be just as important to him, if he warrts to examine'' 
this bias and use it in pi^nning His own work. 

, * IS • 

Management concerns are a major element in determining the 
environment of use^^Choices in .this area are a matter -ot individual 
style, as inuch as company policy: the manager who values the Informa- * ^ 
tion , approach allows, extrA tli^e for it where the manager who Is^ I 
unsympathetic may not. There are other such concerns .which appljj^ to 

• information as a whole, as well a^to the use of pur chased serVltes as 
one means of obtaining it: ^ 

-• . . ' • \ --^ ■ 

- are individual efforts and team work boph encouraged? 

- is innovation desirable or a threat to established lines? 

- is a quick answer or no answer an acceptably response?^ 

- , is any premium allowed for quality of information? 

- are benefits realized in this project expected to help 
^ . support other information services, or vice versa? 

* ^ * 

The maturity of the specific project * bears directly on how 
much information is needed and what type of service will .supply it. 
BreadttTof coverage, detail, and when to stop a^ interactive decisions 
which apply at all. stages of urgency and Justifiably expense. Basic research, 
patenting, and market development may all benefit from a state of the art 
review. A novelty search- is more Sipec^f ic,, once a project has been 
defined^ Prolect negot-lations or contracts must defln&^'and clarify tho^e 
aspects ^of the new development thab someone is^wllling to pay for.^ " The 
over-riding question-^of liability has grown far beyond the cavea^ts of 
business law, in response tc^ government demands by -such agencies as the 
EPA and FDA- Complete eva^luation, beat available technology, and 
envir9nmenta.l impact statements represent a significant part of the 
pre^nt tota^' demand for Information services. . 
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Intervlev/land Qu^tlonnal^es 

Initddl interview^ with t^; major ytf&ers of Information services ' 
confirmed the concept of dl!menslonfr ip selection , factors, and Identifiable 
user populations as a reason for differences in the values they perceive.. 
Further interviews to i<f/ntify addi1;ioital/ variables and their interactions 
surveyed the pattern of /selec.tic^ for specific popui'at^ons of information 
searchers, patent aptqrneys, a universijty library, and the users of a small 
branch library. ' , / - 

A more detailed que^tioniSaice was developed to explore the • * 

charac.teristicB^of ; the T|ser popolatlons Identified, and imprwe ^the 'def ini-* 
tion of apceptabl'e teilnlr. This-^as deslgneS as ap opinion sS^ey for use 
in hand, with written definitions, to decrease the effects of the variable 
bias of the investigator during interyi?Wg,* It Was addressed to^a random 
sample 6f 240 named individuals picked from the professional employees -rft 
ER&E. In this s^ple 130 individual^ were locate^ at the engineering 
research center, 90 in laboratory research anti 20 in smaller groups such 
as information speelallsts or patents/" These sub-sets represent 10-20% 
of the employees, sampled: ^ The list was then checked or a telephone call ' 
was made to ask: have you during the past full year (1975-1976) done any 
searchiiig^i^t the published literature apart from reading current jou^iials, ' 
eltl>er yourself or by asking someone else to conduct a search for you? 
A questionnaire for this survey was sent to a total of 89 isho were known 
as users or who th^ identified themseJLves. This 4-page fprm took an ' 
average of 15 minutes to respond. ^ . ' . -> 
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The initial query shbw^d that 4:9 of th^* 130 -engineers use avallabj.e 
literature services to search for information versus 81 who search only in' , 
company -sources or current journals^ This ratio* is nearly V to 2 and the 
same ratio' for research was 40 to . 50 of^ near 1 to 1. The greater interest 
of t;he researchers in information services is, reflected in responses 
received, which was 38 out of 40 from research versus 32 out, of 4Sf from 
engineering. Similar trends appear in the average number of searched made 
during the year, which was about -2 ,£or tjie engineers and over 4 for research. 
^ The same applies to familiarfty with*':d^f f ereiit Information services; a * * 

check list by title 6f those used at least once during this period showed ^ 
an average of one for the engineers, and three or more for' reseajr<fh. 

Values perceived by these two groups were ranked on k 5-point ^ 
scale ivltal, yes, neutral* not needed and prefer ^tijk The weighted 1 
average was strongly positive for^xeseaVch and* negatlAm for engineers 
* ^pn the use of originals versus abstracts, using known services, and 

asking a known indlvldtial to collect the information*. Both*, groups* agreed 
(90 vs. 80Z)>that it 'la important to get key references promptly, b'efore 
any further search. Both ranked c&mpl^teness ln,a seardh alieadof accuracy, ^ 
the' engineers a little more so. . The^ strongest agreement was on two ques- 
tions with heavily negative response" and 10-26Z of the total as "prefer not," 
for the Importance of knowing, exatt 'dpllar costs ro compare services and 
for the user /customer himself to know how to use computerized 'information i 
w/ systems. * . - , , 
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These replies Bxi In an environment vliere information a 
are available upon requeslf and an abstract service tailor-made to 
interest is distributed iro mX employee?. Without' this bias, it "l^\falt 
that the ratio for originals v8» abstracts would be even stronger f^ 
research. Different rafakli^s might, also be expected on some factors^ 
e.g!, on accuracy vs. ^ompletness. ^Further inspection in this partict^ar 
sample' shos/ed that the only engineers who preferred priglnals/ were 3 ^ 
men out of 33 responfiees who had eome into engineering from ^ backgrov 
in research. / » /. , 



Conflict in Meanings of Key Words 




Le/e 



A Dtajor conclusion of this st.udy Is- the completely different 
connotation/ which iifferept groups of usets attach to -^he same, key 
words. 'A functional ^approach ^ this semantic problem, is^* shown in « 



Table 6. 



/ I 



Timeliness of .information has f special meanug jEor R)fc, namely, 
information produced 4t the right time - neither too early nor /too late. 
This may ;create special values for a redundant service which c4n i^roduce 
pertinent information recently current, but not yet/adequately covered in 
the compr^^nsive^abstract/index systems.. 

The manager of R/D makes a basic choice^ as to the amount of time 
he will allow for "the use of information, frequently about 10-20t of total 
project tlme> as distinct from going out in tW laboratory or the field to 
^et more data. The choice is labor-intens lye, /depending on competing 
values for the worker *s time. It may be changed little if any By a high^ ' 
powered new service which produces mpre information In I6ss time but ^. 
requires pore time for proper analysis. There is a strong consensus drat 
the .time devoted to more effective service^ is essentially subtracted 
from Id^ss effective sources of informatioi 



Priorities in selection for research .are addressed' to details^ in ' 
pertinent -references^ having .a plear s^nress on a definite concept or request. 
The search can stop. when you get th^p^nswer,' or new targets may be defined. 
Dependable qualitjj^ as a resource matetial is a primary concern., ' Costs and 
cost/benefits are applied separately/to each proj^ctv 

.Work flow to keep the information system operating dses lOOZ of 
the, time available,* for the work^s and managers concerned. This is split 
between routine operations and ^i^pi^. TJie operation is labor-intensive, ! 

Lrect %ost£ 



with a , strong component of indirf 



:s to oQsilntaln the system^ 



TimelinesB for operations is measured in tj^rms 'of ^restated time 
scheduled to g^t new items out, ind how dependable this production schedule 
is fbr the user. The schedule times whtch are frequently most tliq>ortant to 
the uset. are the earliest date when he might get the inf orfiwitlon, and the . 
latest date . to which tjiia/ may slip when things ga wrong. 'These can be much 
more significant than tM average time, which the operators ot the 'system si- 
might prefer as a measure of performance. 
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Table 6 



^ . • Conflict In^Sets ot Meanings for the^ S||fee Key -Words 



Parai^eters of* 
Value to the'Usefr 



^3 



T 



for R/D Function 



UsefOl ^ ^ \ 
, Information ^ 

Specificity 

"Haffds-on** access* 



Depth ^s breadth 
Timeliness 



Total time , 
allowed* 

Alttltutfe tovafd 
new facilities^ 



V^ost fcac^rs 



how tojido things, or what 
not to'do; knowledge » 

pertinent to specific target 

•all pertlneiit originals _ 

•details, on^delacted Items 

net too ear*ly, not too lat^e 

10-207. (Info/ vs lab)^ 

cautious, subtracts from 
'total tlmr* available 

per Itetn charged, 
Irfbor -Intensive 



for Infprinatlpn Of^eratlbns 

accessible output; ' 
i work flqw, dolnp. 



4^, 



* cover a^l good sources 
abstracts and references' 
breadth of *c^verage * 
at the time promised 
lOOX (routine + output)^. 

prpmote multiple uses, 
to lower unit costs 

> .justify overall .program^ 
labor > capital * 



•for Planning 


Function 


✓ ' • . ^ f 




target selectionl 


what to do; 


1 insight 




relevant to targjif 


t areas 



digest of po^nts|of. view 
breadth of viewJ*not details 

liiunedlatel7,f on/demand * | 

* 1 
elapsed time to deadline. * 

"*i ' ' I » ^ ' 

I if it provides answers needed, 

yotth what it costs 

total costs: •tfine is costly;' 
capital, >labdr 
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Many of the same parameters apply to the manager of a commercial 
Information service and th|p manager of an in-house information system/ • * 
Both seek to pptlmize th^y.tise of special new facilitiea which Increase " 
their capability for speciaKproJects, and in general encoulrage multiple 
uses for the same item. Their priorities In, the selection ofMjiformation 
tend to emphasize breadL^h of coverage , according tp pr^s'eribed standards 
of output, so as not ,t6 ^ss a good reference^*' If a ghoiqe m^ast be made, 
it may be more Important to cover all soutj6es and not all details The 
work output frequently stpps with refei:ences and. abstracts,. . and. blblfo- 
graphic. coverage may be enough. A preferred criterion of success fpr' ' 
future demands is not, the total output, but much of it d,s actually 
used. Cost justification and cost efficiencies are examined for all 
it€!ms in the proer^m, and cost benefits are considered in general^ for the' 
wholii operation/ rather than for single items. ^ , 

^ Pltopi^^g is strict pn elapsed^^lme, and ignores or places a heavy 

discount on Information, received after the 'decision has been made. Back- 
ground information is selected' for relevanpe, a, traceable op logical 
connection to the area concemed^ and cannot stop with pertinence to a ' 
"single point. The planner will use whatever sources he needs ai\d can 
find within tl^e time allowed, knd often has well-developed sources for 
Information outside >the published literature. Within this time frame; » . 
costly special senr^Ba may be justified as a capital expenditure, *wotth 
what it costs. r ^ • V . * 

Timeliness of information* for ^flanning ,i^ measured in terms of 
speed, to meet special rpcjues^s^ This involves current operations. It 
ftequently benefits from pr^aviously prepared document collections, vertical 
files, or abstracts preselected for browsing in areas of interest. These 
are available for immediate use on demand, to seek information relevant to 
projects being planned. jSpeed is essentfal" to relevance. ^ . 

Priority, in the selection of useful information is addressed to 
overview, not details. The jilanner. Wants a collection of different. 
points of view 'for looking at his problem, and is particularly interested 
in methods for evaluating the information available. The potential value 
of a ^Ivtn %formati-^n service for futar^^ reference dfepend's in part of 
the reliability; with Vhich it /can be expected to .present a specific p6int ^ 
of view. Repeat business t^icall^ requests the work <?f a known ^ 
individtia;. Th^ planner is extremely tonscioufi of total coats; time Is 

™i "^^V ^^^""^ ^^^^^ project. Long-range plaimdng towards 

.areas gf future Interest rather- than ImmecHate projects iay be less 
stringent on .the /time scale, but, is equally concerned with the evaluation ' 
of sources and points of view. vaj.u«i.xuu 

The principal point In Table 6 is that the. differences observed are 
•linked to function: they are not completely random as to the individual. 
Thus, relevance and over-view tend to relate to planning, where pertinence and 
detail are of more interest in research.' These linkages can help' to reduce^ 
semantic confusion. Note again that- in this comparison as eJLsewhere we are* 
considering^ only relative values, not absolutes. . ♦ ^ , 

, Function as here defined has to do with the structure of language: 
information operations are concerned with doing , the preseht abtive participle 
The research/development ^function,*is concerned with hbw to do, the art or 

• • ■ ' \ ^ 

xvll 



deve}.opin^ 8tate(of the art of the patept dttomey^ Planning' fcanagei4Mt/\ 
is concerned with target /evaluation or what to do, which^can be considered 
the subjunctive mode, of riiight or could or should , . The engineeir-who con^ 
\ siders himself a ljuilder and pot a researcher may prefer ta nief ine the- 
three '-^iipet ions ^is scientist/engineef /manager rather than research/ope*a- 
tions/pl^ni^ig; -and the table can be read just as well with these terms 
as co||jBnn he^ings. fh6 differences observed ap^'ar to be real and 
characteristic, regardless of which labels hx^ applied, > 
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Applications of th^ Functional Approach 



In'pra^tice, this approach has helped to defuse areas of apparenr 
conflict in assessing information ^ne?eds. Specific exaipples ar% cited. 
Junior staff could see that a nbnagenient change, in decision based on 
additional aspects of informati^dti may t^Qt at all represent a lack of 
confidence^ In a amail business situation, management could allow for 
tfhe information priorities of both scieritisps aiftl planners, corresponding 
to their x^kl differences in function.' Tha patent attorney has a,§trong' 
component of planning^ (targe^t selection) in the use of fnformatiqp, a6d the 
patent searcher he li!ke&^ and trusts is the/^one wfio gets his business. For 
an abstract service, opinions of the searchjer on the larportance of * 
access to originals may be easier to obtain than those of "the scientist, 
but they are^ not 'the same. For a' special library *wlth limited staff 
serving all three ^functions i (research, operations, and ^lannlng^ a miirked 
inrprovement in the service ^to one may resujlt in a definite loss In' - 
service to the others, unless an improvement can be made in them as well. ' 
In thlsxase a .review' of user values has identified a Vnon-computerizable" 
skill /as the ability of the staff to upgrade the input to the system; by 
checking .the originals of documents selected from the incomplete infonna-N' 
tion f,ound in titles an^ abstracts. 

4f This* study is concerned wtth the value preferences of the user 
of inf.ormation Services, the individual scientist or engineer or 'person 
from other papulationfii who wants to use the published literature for more 
than current reading. It is not'-dfrected to 'those who don't care. Most 
'Scientific and technical information systems are not used solely for 
research or for planning or for service funciioilff and moat users are not 
pprely rational pi> subjective or jud^gmental. The person least sensitive ' 
to the co^t effectiveness of dimennions in valud other than' his own may be 
on^ who;thinks his sydtem could become all things to all people,, if it 
could only get ^n6t>gh support. The selection and optimization ^f a service 
requires an awaifeness of the varied wqys in which it will be valued by dif- 
ferent users. Differences in points of view may be npre readily resolved 
when they are recognized 'as miiXti-dimensionai,, and not necessarily as • 
ogposites. , * • 
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1". INTRODUCTION 



'The rapid expansion of in^ometion reefearoh services has created 
an industry that tends to be dngrovn, with usets. and producers wHo are their 
own be.st customers. The number of such services Is well over a thousand 
and continually growing. Many or tt|ese services are directly^ competitive, 
and it is increasingly difficult for the potential uder to know whl^f 
service to select foi^^a specific use. ' • 

"•^ 

This report Is a .study* of the selection process to .which different 
groups^of users decide to buy or use different Information ^N^ices, for 
searching in the published literature. It has been conducted for .the 
National Science Foundation, Division of Science Information (formerly 
NSF/OSIS) under Contract Number C-1027, aa a part of t*he larger current liSF 
project on "Improvements in the Dissemination and Use of Scientific and 
Technical Information (S'fl)." ; ' 

The postulate proposed and developed herein is that there are 
-'identif iable groups of users with different types of fieeds who differ in 
how and why they chocCse one tyt>e of service (Sver another, in STI and 
related fields. The characteristic. v£^lue systems applied by different 
users in this selectiori process are not the same, depending on variables 
which can be defined by dimensional aiu^ysis. The results obtained confirm 
this thesis, with the subtle but significant difference tHat identifying 
variables relate more to us^ as an, event tha^ to the user as a^ Individual. , t 
' . ^ , , * __ - 

A matrix constructed to represent these ;vari^les an^ patterns 
ifi .use provides a convenient means to explore th^ir similarities, differences, 
^hd linkages between them.. This matrix of user-value systems was developed 
on the basis of individual interviews with some 30 regt^ai: ^sers* of. purchased 
information services in various divisions of the CJpntric4:^, Exxon Research 
and Engineering Company. It was then tested and refihed Iry, Interview^ 
with an eqtial number of managers and users of information aervisbes in other 
companies. ThJLs .checking with outside sources was stiptilat^d in €he project 
scope of ^^k, as an, expanded program to disseminate the results obtained. 
A questionnaire was developed and tested at the end of the study to examine^ 
.the acceptj^i^i^y of the parameters selected ^nd the definitions proposed 
for them; Preliminary test results and suggested modif ijcations are discussed. 

This is the first-year report in a proposed twp-yrar project. jThe 
objective for the second year will be to examine the factors Involved in the 
success ojE systems and services for information analysis. ^Interactlpg 
relationships to be explored Include the effects of location Cthe itfformntlon 
analysis center), of personnel (the analyst'and the us^r/customer) , of 
pro^dure (the analysis) , and personal lajacts in bo*th directions between them. 
The plan fo^ the second ye^r includes individual Interviews and. case histoties' 
of^a nmber^ of such systems. ^ « 

As a matter of policy/ no id;lntif ication is publi^ipd bsreln of 
the exaoT sources of the individual oplrii^on^ and experienc^Fpeported. 
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»^ 2yJ,bEFINITI0HS . 

■ ■ A •■ ' . 

The purpose of .>thia study is to provide a guide for the user of 
scletitific and technical Information services » to help him/her* identify 
his patterns of use and needs. It is addressed to the user who has decided 
to im^e8t time or money for. Information services, and who wants fo select . 
from all the services available -those whi civ will .meet his own requirements. 
1 1 explores the value^ systems observed by. different groups of users' In .^e 
proces^of tfuch selections, so th^ each can identify others whose points 
of view are similar o^ .different, to benefit more directly from shared 
experience. The study excludes by definition a choice based on field of 
coverage alone: . if there is only one service that covers a given f ie^d 
there is no choice, and^it gets all the business. Usually, if not always, 
there is some alternative route,, and therefore a choice between competitors. 

• . * * ' * ■* • ' 

Thfir literature on the evaluation of iniformation systems is beset 
by semaatic confusion, ,wlth cohflictlng or contradictory definitions for 
the same key^words. Much of this literature h^s been addresse'd to the design, 
and performance of a given system or service,, where our concern is the 
choice between services. It ijp necessary for the j>urposes of this study to* 
give cl*ear and Arbitrary definition* to a set of l^aslc terms : 

The user we are considering liiay be anyone who is selecting an 
information service; Ke may be a scientist, an engineer, a library 
searcher, or an information analyst. In centers where these services are 
readily av^lable, JLt_m#y be unusual for the g ame perflon to be both ^'user'^ 
and customer (the one who is asking, for information); more often the "user" 
is collecting information at least in p^t for someone else. "fiMrchased'*' ' 
requires that the service be commercially available and therefore viable, 
on the market -either as a service to order or one readily availaBle to many 
users. Such a service provides processed information, which saves the user 
from having to dig it directly out of original- sources . Our study is 
directed dpecifically to scientific and technical information; similar 
principles may apply to'other areas of knowledge with different peram^ters, 
e.g., for time or the nature of* data. ^ . 

Types of Service * f' 

Purchased services from which any user may seleVt fall into three 
major categories:'^ ^ 

access tools which the user can use himself, at all levels 
^ of complexity. These include indexing and abstracting services:, 

computer data bases used for routine retrieval. 



With due a<K>logies for the .English language, "he" will be used herein** 
after to include "she" or ."they"; "hers" ii included with "his" or 
Ztheir"; "himself" equals "herself" or "themselves", etc 



24 



- paid searching for known facts, where^the question* is 
defined before the search begins. 

^ a service for Information analysis, where the user pays 
someone both to find and to analyze the Information' ' 
available. • 
*■ * ^ ♦ 

The dlstlncti6n here Is that "searching" the quest±oh,ls 
pre-defined. Definition may be a lengthy process, but any* change In' the 
'question once agreed upon Is considered ^s a new search. It Is also only 
^ search for facts, or references*, not the evaluation of facts. As a 
corollary of t)fls> a commercial service which does such searching may • 
state that any searcher on the staff, with proper training, can ajjcept* 
an Incoming question and come up .with the same **ahswer . - The customer for 
a search may get all the orlgln:al literature* aiid references, or he may 
laake selections^ from a se^^rch report (bibliography) and request specific 
items. ^ The customer does his own evaluating, either way, or arranges to 
have it done as a separate step aftef the search is completed. ; 

' ' "Informatlpn analysis" Introduces a new set of interactions that 

go beyond searching: the user continuously redefines the question by 
ianalyzlng the. information being obtaMed. "What is*..?" can be a simple 
or complicated search; "What if..." is quite different, and requires a 
different level of personal or staff ability and experience. This may 
involve conferring with the customer, or the user may do all the analysis 
and supply ^only a digested product. The same information -services, even 
t he m ost sophisticated bnes^ may be used eith er f or searching or fdr Infor- 
ma,^lon analysis: the difference lles^in ^the. user, and how the information 
t^o^s and services are applied at ^thls specific time. In any. case, the 
information analyst Is applying a hdLgher level of skilly tha^i in'' sear^chlng 
alone, because the analyst must understand the technical language of fhe* 
literature and the customer's needs,^ as well as the inner workings of th€j ^ 
searching procedures used. 

- ' ; f 

The value pyst^m which the user applies can be viewed^ as a setN ' 
of interactions which determine what group of lactors he considers most 
laiportaDt in his selection of services.' Principal parameters in the 
decision process are the selection factors preferred, the type and function 
of use, an4 the type of services. 

Selection-Factor Dimension^ . 

. *^ * ' 

A dimensional approach to this complex system/ is recommended. 
There are a large number of discriminant factors which can be considered / 
significant in the selection ptocess. As a ^irbt step in this development 
these can all be categorized in t^ree 'dimensions, as quantifiable, 
qualitative, -dr judgmental. This type of dlAenslonal analysis is based 
on previous studies of multlvarlant systems fot the evaluation of projects 
in research (see Appendix A) . The idea is to group variable^ which are so 
linked that they tend to move together in fny' change or comparison. 
The list in Table 1 suggests that* cost, time, physical facilities and ^ 
personnel can be considered as one such group, in the dimension of 
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-Table 1 
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DimensiOTts in Seleation Factors 
(Aside from Area of Coverage) 



Quantifiable 
(numeric) 



Co«^r- Direct 



Qualitative 
(ranking) 



r- indirect 



, Management Judjpent 
(yes /no) 



' Complelbne^s 
Time 



Format 



■flexibility 
Responsiveness 

Feedback * 
Feelfng aspects 

R^utation V-i 

Viabilit:^ . 

Consistency 



Cost/efficiency, 
Cost/benefits : 

Fields, nunibers, Z 

Entry into text or 
jlndex 

Retrieval speed 
Timeliness 

Appearance 
Uniformity 
Convenient access 

Jtepth can be varied 

Change priorities, 
Accept suggestions 

' To/ from user 

"I li)ce it" 
Public relations 

"Our way of doing things" 
Past recommendations 

Will, it work? 

wiu it last? 

j 

Dependfitble bias i 
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quantifiAle factors. That is- to- say, any circumstance which has a majot. 
effect on^cost, time, facilities,, or ataff requirements is likely to have 
some effect on other variables in the^'same grpup. 

The background of Table 1 is explained^ in Appendix B- Initia^, 
interviews with a few major ui^rs 6f Information services developed a list* 
o¥ discriminant factors which ^tHey consider as practical criteria in select- 
ing between services » prelerre^/ Examination of these lists showed that'thfe. 
factors tabulated were all quantifiable, but there wa^j^ always a column of. 
comments which were not/ Entries by differejfc users fox specified services 
showed quite goodf agreement for the quantifiable factor's, but not in the^ 
commsjats. These were different for each user, .or ^r the same user for' 
different uses. ^st«of the comments refer to a^^ond dimension which is 
well tecogiiized .as' the whole gamut of qualitative factors that canntSt be 
measurg^ exactly, but which can be handled by the technique of subjective 
ranking. The exact names given to different factors or relationships in 
this group are harder to handle than the fact that so many of theS tend- to 
be interrelated, and to iaove together when: any of them* is significantly 
changed. . Differences 'and relationships between Quantitative ^and qualitative 
factors are frequently recognized In the design and optimization of infor- 
mation systems. One such statement is that if you can completely structure 
the input to the system (the definition pf user needs), then you JPan structure 
the o^p;ut (what jtype of service or report will meet these needs) and the 
qualitative aspect t^nds to vanish as a variable (see Appendix B). Stated 
differently^ if you can give the customer a product he likes and wants to " ^ , 
use, he doesn't care too much about other^ aspects of quality. " ' 

^ The decision process is not based just on fact^ and personal 
pr^erences, however, w-xhere is an elusive element of judgment of "common 
sense" which is not -^ither quantitative or qualitative. • Eloquent testimony 
on this can be dfterkd by many a development engineer who is baffled when 
management has turned down a pet project that looked good on paper. Manage- 
ment judgmeHt tends to^e a ygs7no answer. This judgmental dimension is hard 
to argue with; it is basVd on- neither nt^merical facts nor subjective rankings 
alone, but on a blend of theae with intuition, based *on experience, ^he 
evaluation of reputation,* suitability or future prospect^ often involves a 
major, contribution from the judgment of othet m^pagers*. It •-is hardjfer to • 
define and is less 6?ten discussed in the l^tdifature on inftjrmat ion systems 
tlftn either cost or cost/gf^iciencies/benefits^, but perhaps this is only" 
because it is so often deferred to others and then taken for grafted (see ' 
Appendix B). * ^ - 

Quantirative and qualitative variabUes can tje characterized as 
ratlapial and subject^^e. Mt6^Ste9nsul-efation.;wJ>th behavioral analysts; the*- " 
'dffmeHaion of businesffSjuidgment wasr recognized ks correspondingly "hituitive." 
The dAnensional approacnS)n this basis has tlje advantage that the ciategoMel 
naped a?e complementary, -not opposites; It is therek^less likely tb leave 
holes In the Matrlpc .of values and interactions. , ^* y * * ^ 

• ' * * ♦ » ♦ 

The list of selection factors in Table 1 is by no mean^ all- 
inclusive. It carefuUy avoids some obvlou^ words such as "reliabilit])^,^' 
which xould be put several places biit.with quite different meanings. Many ^ 
disciiminating factors (such as cojnpleteness or accuracy X^ljbve eff^ct^ In ' / 
sevei'al dlmensi(yi8. As these effects were examined more' critically durtag 
the iJtud^, it appeared that the special meanings or Implication^ attached - 



to key words (such as timeliness or intended u9e) are a characteristic part 
of the distinction between different groups of users. The starting point 
Is/^that dimensions in the selection process can be recognized, regardless 
of the exact names associated with them, and that they can be' used to 
characterize parts of the selection process. 

" . ^ * . 

The literature on tfie evalttation of infoijuitlon is biased, at 
this point. Part of the probleiais seteantic. As a point of departure, 
each author tends to invent some ncTw word - or attach a limited special ' 
meaning xo axioli one. Such articles are often contradictory, reflecting 
strong individual points of view. Much of tliie literature is written- by 
the purveyors "of information services who ed^hasize their concern with 
matters of cost, or by system managers concerned with the ^ue'of time, 
■yhese groups agree wit;h each other on the Importance ,of cost, however, 
although oth«t osers^may not. There is a basic reason for this: both are 
in a senM^endots must sell their time and services to someone else 
to stay in business. ^ 

* ' . y 

This information-evaluation literature seems to have started with 
heavy emphasis on factors of cost (1), changing gradually over the yeaifs 
to include cost/effectiveness and cost/benefit ^factors (2-7, 8). These 
are progressively more difficult to quantify, because they involve subjective' 
elements (9). -Many authors have recognized and bemoaned the problems of 
evalu^tin& ^complex v ariables- containing both obj^tive and subjective elements 
which cannot Be exactTy measured (10). The total number of factors recogniaF 
able ^ . the sel-ection between services can be expanded almost at will: a 
recent ch^klist recognizes 144 ef them, orider 7 different headings. Many 
i«.of^ the9e are closely linked, however, and they are not stated as Independent 
i variables (11,, .12). JudgmWtal statements t^nd to be kept otit of the open 
.iJ^rature or to be phrased vaguely, in general terms, regardless of^how * 
|lmpprtant they may be in the decision process '(10.). 

"7 ' • ^, ■ . :^ 

\ User Populations j 

Identifiable groups of users are found to differ Sharply in the 
value systems they apply, in deciding what services to select for a given 
Ude. Jhe grouping of selection fdctors into dimensions tends to be con- 
firmed by parallel differences in their appeal to dj.fferent user populations. 
This- study started with selected heavy users of scientific and ^technical 
information and services for access to it. Those interviewed included 
scientists, engineers, patent attorneys, information searchers and analysts, 
planning staff and managers. These user populations can be p6nveniently 
grouped IntTo three categaries, 'as suggested in Table 2, on the basis of 
whether the information they most frequently want to have "hands-on" for 
their own use fs original documents, selected references (by title or 
abstract), or evaluated overviews. The identifying characteristic proposed 
to distinguish broadly between the* is their ^uncf ion in thp *use of 
infopnation: for R/D, for Informatiotk oper^ions, or for planning. 
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Table 2 



X 



User Populations 

Research Function 
Scientist 
Engineer ' 
Patent Attorney 



User Population Characterist 



Typical Preference for 
"Hands-On" Access 



^11 pertinent origitials 
representative orig?nals 
highly selected originals 



Information Operations 

Searcher 
^ Information Analyst 

Planning Function 
Planning staff 
Manager (planning) 



references requested 
selected as relevant 



all viewpoints 
overview of field 



Special Concemg 

reaspns -^r difference^ 
data, desi^ details 
exact wording 

\coverage pre-defined 
optimized during Search 

> 

eval lia t iw-^e t hod s 
reliability of .elements 
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Users concerns primarily with the research function include 
scientists, engineers, and attorney^. These are professional men whose 
Information need Is for origin^* documents, althpugh for quite different 
reasons. The scientist frequently wants to study and compare different* 
sets of original data, or the 'mfetliods. by which they were obtained. 
engineer usj^lly wants a single* set of reliable data, not all the de^^^ 
that there<^e. The attorney 4rafting 4 patent apj^licatioil^^irir new 
agreement needs the compifete and exact wording of selected originals, 
but is, less concerned about data. * ^ , 

m 

* • Differences between engineersc and scientists in their usage of 
information services were noted 'in InitiaJ. Interviews, and confirmed in 
discussions with a number of information managers and librarians. Statistical 
data /in sjipport of these results were then obtained in the preliminary j ^ 
re^uft€Hatom an internal survey hf questionnaire (see Appendix C). 'The 
typical engineer tenfls to rely first on Information from handbooks or from 
colleagues, and particulairiy from compytny reports and correspondence. 
When he asks for a literature sdtrch, he is usually satisfied with seeing 
whatever selected references are readily, available. TJie same tendencies 
appear in both engineering technology and in engineering design. These . 
obsetvations are fully in line* with previous studies in the literature (13). 
The tfe search scientist also wants his references as quickly as he can get 
them, but he is often willing to wait for days or even weeks to get * the 
last article on his selec6bd list. There was a unanimous agreemeni: among 
IP engineering librarians that* this distinction between the typical acientist 
« and the engineer is real. , ' 

"In information operations, the user- concerned may 'be a paid 
searcher oy analyst, or the manager of ^'system prpviding, such services. 
It is worth repeating that this^study is seeking the viewpoint of the . 
uge^^ of services rather than tVie producer or vendor, even though this may 
be another fun^tio n of the same individual . The searcher (the individual 
who is acting as a searcher and not as a sc^ent^ist^or as a manager) is 
frequently satisfied to stop his search with an abstract that identifies 
useful source documents; he may ilot re.quire their physical production, but 
leaves this choice^ fo the customer. This difference betwe^ the viewpoints 
of the searcher and the research worker has a significant corollary in the 
value of access to the original .^The searcher is satisfied that he has 
added a valid reference to his reSbt, but the selected reference which 
cannot be found after concerted effi>rts has a negative value to the customer 
fot wasted time'ahd effort\ The absteHcting or indexing service whose 
"customer feedbadc" comes mostly from* searchers will not^get this reaction « 
as' strongly from them as it would from the' chemist or other cus$omers 
(see Appendix C). 



\ 

rlannlng Is^a function of managemelft ^hlcb is oft^n delegated 
In varying^ degrees to others. Outlines oS^ollt)^ for the selection of 
targeta ^y .be prepared by a special planning Maff» or by other levels of 
The planning function InvolXres spedlal sources of Infonoatlon^ 
often uj^vrltten, for such matters as organization policy or future pros- 
,pects. / These Include questions entirely different f¥om. either the 
quant^ies of costs/budgets or the subjective 
relating closely to Judgment and experltoce. 
with/the services to be selected » the^^oanager 
ii^oulra jely on other sources of judgment. - The 
analogy, considering ^thq opinion of^-another lliinager they can trust, or the 
reputation of the producer as an individual or as an org^ization 
rganization" in, the^ generi,c sense includes 
liversity, agency, or corporation involved in 
technical information. 



qualities of perfomarice, 

:king personal experience 
interviewed said'*that' they 
suggested reasoning by 



any enterprise^ such as a library, 
the processing of scientific 



Planning is concerned Vith the selection of targets fortfuture work. 
The planner is typically pressed for time, aifl more interestjed it the moment 
in a& overview or digest of information than 
find them. He wants, kmed late access to releWnt ideas, available viefwpoints, 
and suggestions on how* to ^ evaluate fhem. His | problem is quite analogous! 
, to the hun^^er selecting wl»S^eapons %o take to hunt for squirrel^ or 
partridge or deer, and selecting a strategy to suit ^he target. He has 
seconda^ ^but. vital' choices .such as^to what to do if he comes across a 
bear, a^d JDbf. most^ important actual choice he may h^e during the day is 
trtietherko quit looking for deer that aren*t th^re and pick raspberries. 



- - ir - . " ♦ ' ' ' 

* 3. MATRIX OF VALINE INTgRACT^IONS . 
* • ' 

• - s ' 

The Matrix diagram presented in the Summary involves a,coniplex 
systejii of variables ,''intiMfactlonte, W changing values which different^ ' - 

, users apijly in thdir^selection o| setvices. . The matrix ^al)pro4ch h^ps • 
to indicate thAt certain patterns, Ind ^.Ittkages li^'t^he user's choice are ^ 
^edictabie.. The pusj)/pull effe^l^f f elated^ fac-tors Within, a given 
dimension has been discussed: ' ptAh either personnel' or /acllitles and you ' 
affect time or* costs ; ;t)ush either reputation- or suitability and you affect 

^ future prospects. The Matrix rows j)re^nt ^the^maj^r varlat>le8 or elements 
•deflned^bove; dimensions In selection factTots, user populations, -ancj types 
of service. Matrix columns repr^senf how these variables are defined, how 
they ^e.percelvfed by the us^r, anS how they interact. U^er/f unction Inter- 

- actloM appear ±0 the different pifLorlties In value assigned to selection 
actors in each dimension; user * interactions with Ihe event or environment 
Of us^ appear as concerns of the user, i^anagementy^nd the specific project. 

: * The Matrix dlso l:ielp8 tQ emphasise that -the concepts involved are 
all cpnsld'ered as^elatiye, hot ^bsolute, as preferences in d d3mamic situa- 
tion. The devel Client; of the a^trix/is outlined Ifi Appendix R: interviews, 
Jlnitlal analysisy preliminary conclusions modified, tested and redefined 
amdng different -dsers and managers, so as to be acceptable to each. Pre- 
;imlnar^ .results specific to scientists aqd engineers Were augmented and 
confirmed'by a user-survey and questtidnnaire, whose development and analysis- 
are outlined in App^endix C. * ' ^ 



Priorities In Selection Values * 

^ ^ , A simplistic statement can be made for any user who ha/d a Choice; 

he wants the best service that he can. Afford for his use. Diffferences V 

ari^e when the individual as user Is forced to consider what function he 

I's ^serving at Ae time, and what selection factors he cqn8i4ers most Important 

for that specific use. ^ The searcher aftd the scientist differ, and neither 

of them gives first value^to the factors of cost that &eem so infportant in 

the literature. Value preferences expressed in interviews With different 

iiser populations suggeste^d the following apparent relationships, which ate ^^^^ 

summArize^d by function in Table 3*. . 
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Table 3 



Value Priorities Acoordlng to Function of Use 



^ «^ Dimensions 
of Selection Factors 

Quant if iable, tfumeric ' 
<Co5t, Time, 'Facilities)' 



Qualitat ive , Ranking 
(Subjective, Convenience) 



Judgmental, Yes/No 
(Intuitive, Experiential)' 



Apparent Priorities 
in Order of Importance 

1 for information operati^s 

2 - for planning function 

3 for research function 

1 - for research function 

2 - fo^^perating function 

3 - for planning function 

1 - for planning, management 

2 - for research function . 

3 - for information operations 



The scientist/engineer chooses first of all the service 
, he finds* most conveni^tlt to use* He accepts the judgment 
.of a manager as to vha*t systems to consider, but he pays 
relatively little attention to quantitative comparisons 
of costs or facilities as long as the systems available 
give him the ansvers he needs* 

The planning staff or manager makes a first choice on ' 

whether the service in question is reliable, likely 

to *stay In business, or "suits our way of doing things/* 

The planner tends to consider budgets on the quantifiable 

factors overall, and may ^^lclude qtialitative factors 

only after he has ruled out services that he finds 

unrelj^le or too expinsive, 

' There is a considerable body of literature on operations which 
.Implies that all other values can and should be linked to cost, 
time, and the quantifiable factors a service manager or vendor 
must control to keep his customers* Qualitative factcnrs may 
be recognized in cost/benefit studies, but somewhat grudgingly 
^because they cannot be accurately measure^* Judgment as a 
separate dimension is well recognized in discussions on business 
managemen^t, but not so clearly in the literature oq info'rmatlon 
services* 
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It must be>eiDpha8i2ed phat all of these' factors v^re recognized by 
all tlie user groups, but ±n an entirely (Jl^feijent order. The copt of an 
inforuu^titJh service, for, example, appears tp be less significant to tKe 
research scientist than convenience or reputation: ^this is not* to say that 
costs can^be ignored,' but only that the scientj»t is more likely than the- 
vendor to put other factors first* » ^ * * ' 

~t ' • * 

The order of priority Is clearest for each function to what choice 
CQmes first, and secondary choices may vary. The information analyst, for 
example, carries out an operating function in afearching and a research 
function in evaluating the Information fomd. The patent, attorney is - 
assiduous both in searching and in planning, not necessarily at the sane 
time. Earlier dx:afts/of Table 3, named user groups for 4ach function and 
ran into difficulty in wh^fe to pUce the^title "manager*'. It seemed first 
to belong with "planner," but on^rther thought it could belong just 'as 
easily with the searcher, or, tlte manager of research.-* This forced the 
realization that the manager viewfe thi^gs differently when h^*is concerned 
with the function of research, or information operations, or planning. 
Furthermore, from this viewpoint, the priorities of the manager and the user 
worker are mucb the same, for a given function of information services* It 
is the function, not- just being a manager, which appears in Table 3 as the 
identifying variable. , 

Environment of the Use-JEvent ' ' • ' . 

' The environment of use is part of the dynamic Matrix.' Personal 
impacts, ma^agement concerns, and the nature and maturity of the .information 
project in teract# constantly with the user in his selection of services. In 
general these interactions enter into the Matrix as parameters or operating 
principles, which may help explain to the user why his priorities in 
sele-Qtion are what they.^re.- Appendix.A includes a backgrQund discussion 
.on the overall impact of rapid thanges in technology. . Subheadings which 
illustrate these effects are listed in Table, which presents a matrix 
expansion index. This shows the relationship between the sumpary table and 
additional details presented in the text Tables 1 to 6, for vatious parts 
of the ^trix of values and interactions. * • 

Personal impacts of the vorkinfc environment on the user include 
limitations and positive effects, both direct and indirect. , Freedom of 
choicyTjetweefl services, inay be sharply affected by limited authority for 
coft.6^pprovals, if it enters into a restricted 'area within which approval , 
by someone else is required.^ \ • 

♦ 

- Within the user's limits of approval, the *'free vs. f^ee" - 
argument constitutes a has^ard when it imposes a forced cost 
decision on users whose value system wp^ld<always place con- 
venience or quality first. Their skills in the budget/cost 
' ' area may be so poor that a forced choice is arbitrary'- or 

capricious, Ignoring- cost/benefit trade-offs that they would 
normally defer to someone else. 



- The restricted area wher^ 3 second approval is required y 
tends to repress' rather t:han encourage the use of "fee" 
services — all such service, not only those which are 
most expensive t ^ 

%J ' ' • ' • 

. - An alternative which does not discourage use of the « 
proper infonoation services is to define some area, 
^ starting with free access to the library, within which 

the professional participant or employee c^n /select . 
which service to use. Each operating system draws ^ 
its own lines between services that are free, or easy 
to request, and those .that are charged to a -group or 
* to a specific project. ' - , * , 

'Another personal concern is whether the informatio||i worker has 
himself the skills and facilities for direct use of the intorm^^tion he 
supplied. •This does not mean that he*^is going to use the .information, 
and it is not the same thing, as knowing how to use the service. Personal 
experieoce and capabilities haVe a direct influence on the ease with which 
the worker can; find and select information he understands (14). This applies 
particularly to the user /worker who Is doing a ^e^rch or information analysis 
fo'r someone else. Two* questions 'of this type are related:* 
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- Does 'the searcher or analyst know how the information will ^ . 
be used? 

- Does he know what specific information would-be most 
useful^ if. it could be found? , 

These skills can* either enhance or linit the value of the information selected 
They need not be actually ysed, if they' are familiar ^nd understood, Such 
informatfio;! can be processed very quickly and fed, back into the system, to • ^ 
define a desirable new search or a* new approach. in analysis. Facilities 
,for testing. do Qot necessarily mean laboratory equipment or field, units; 
many types of inforjnation can be tested out in other ways, say.by» computer 
modelling or by discupsion. * ^ \ ^ 

Personal interactions include the user's evaluation of the aspects^^ 
of information handling in which he feels th^t^, he is usliig hi,s own highest 
skills. This has two effects: in such an area the user tends to/choose the 
^ services of a sp^ific individual, whom he feels he can trust, W he may 
. want to reject any such help as undesirable interferemce and dcw^this pa^rt 

of the work himself. The aggressively individual ^reseafchewi;;;^rt^Srej^^ 
^any analysis o^ original' references by someone else in his areas of special 
interest. He prefers to find for himself any research leads or holes in 
the data. The planning manager may consider one of his essential; taska as 
browsing. in selected sources to find potentially useful' information from 
results in anotjher field. For any user, thfe Impact of paying someone else 
to think for you in information analysis may. mean that you want t^ know 
exactly who is doing the work, and. to be notified at least if the individual 
axialystnis changed part way through*the search^ This personal Impact varies 
with the type of service, as. indicated in the MaK4x>^ it is higl\ for .infor- 
mation analysis^ and low for routing access, todls. 
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'■ Table 4 

. Matrix Expansions 
(sfie Matrix, Sumnary page Ix^ 



Variables 



Selection Factor Dimensions 
(see Table 1; B-2) . 



User Populations 
(see Table 2) 



Typea of Service 
(see Definitions) 



User Functions 
(see Table 6) 



Interactions 



Users* Priorities 
(Table 3) 



Environment of Use 
'(Table 4,5) 



Interactions with Environment of Use: 



Personal Concerns of the Us^r-worker 

approvaJL atithority Hjnitations, r'epressdve effects 
forced cost decisions; areas of free choice 



experience on how 
what wexild be hp 



tjl i 



information will be used 
eful; immediat>e fe^d-baclb 



areas perceiyed as own highest skills ^ 

reject help or choose the person; broitsing; analysis 



^ n^w Lechx^ues as a status symbol; economics 

expandi%itilization if^ cost-justified; operations vs. research 

vulnerability to negative data. or wrong information ' 

research can beneff it planning less strict. ab to source. 

» 

Management Concer^ns i ^ ' ' ' ' 

partly matters of Individual style,- as mQch as company policy 

total time allowed for inf(^rmati(^ vs. other work 
eithfer individuality or team effort can be overdone ^ 
is innovation desirable or only a threat to present lines 
any premium allowed f^or quality of information, or special areas 
debits for alternatives^ quick answer, best atisw^r^ no answer 

Project Maturity • ^ . \ , 

how essential is it to get how much lifformation / ' 

types of search: for background, project definition," R/D^ patenting 
evaluation of government agency requirements, publications, proposal 

'itepth of search: breadth, detalj^, when to stop. 
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The Interestld uselr finds ascertain satisfaction in knowing and 
•using the newest and most sophisticated technique's, and is likely ^o invent 
reasons for using thein as part of the learning process. This is valid if 
the purchase or availability of the service selected has been justified on 
other grounds, not just as a pretty n^ew toy. The experience of firms whittv 
overbought ' on computers is very much in point. This hazard may be reduced 
y:j>tryiiig the basic cost justification for special facilities to actual 
extent of use, and allowing only incremental valu^ to n^ furibtions that 
• are expensive but seldom used.' * * V 

The time and costs whictl dif fereirt managers are -willing to allot 
^. ' ^ , to special new service features may differ radically as a function of use. 

The vendor-manager wants to find new features that he caii add to his service 
as selling points, at minimum production post* to make his product sound 
unique to potential customers. The manager^f research or pq^odUction will 
not resist if this'meets* an unfilled need^such as a^fail-safe system for 
patent problems. He does not ,want to keep paying for a spec^l feature if 
, , * It is rarely used,' and he will* guard carefully against wasting budget 'dollars 

or the tike of his people to Deceive^ or process information' that they don't 
need. ^The operating manager of the inJEortoation system is^in between: he - ] 
seeks to maximize or to find multiple .uses for new features in the systems 
that he has bought, since they increase the overall range and flexibility 
of his operation^. % 



Completeness arid' accjiracy may be taken for granted in a service 
where performance ^stand^rds are maintained. This is not always true and . 
where it is not, these parametfers quickly become vital. Vulnerability to 
poor i^ormation or negative data from the literature varies with the user 
and the specific use.* It is high for any quick answer, which is -taken at 
face value. Wt\)ng data can be a significant factor in the deputation of a 
service, downgrading jiidgments on reliability for future purchases. This 
is 'not the same thing as negative data, or even unreliable results which * 
are jiroperly reported. The research planner may derive definite advantages 
from detecting a result which seems out of line compar^ to some, observabli^ 
trend or average, and in exploring the^ reasons for this discrepancy.. 
Uncertainty or -negative date can be an important part of the total picture. 
This can be a nuisance in searching and even more so in platining, unXess the 
uncertainty itself is in^ortant Enough to become a target for sjtudyT 



A definite lack o$ concern for the scientific quality of informa- 
tion ai^plies to certain operations of the planning staff. The planning 
manager, seeking a quick overview of the information available in a field 
new to hiin, wants the videst variety* of points of view which can be applied 
in evaluating the situation.^ He is not overly concerned with whether a nev 
viewpoint comes from a Nobel^LaureatVor a tabl()ld editorial, since he is 
trying tp look at. things from all angles. The tabloid bias may be just as 
important to him, or even more so, if he wants to examine where this bias 
came from and what if any ting can be done about it. 
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Management concerns are a major element in determining the ✓ 
environment of use. Each manager makes a basic choice as to the Importimce 
which he attaches to' scientif ic/technical information ^s a whole, from \ 
♦any source, compared togoing out in the shpp or the field to get pew data. ' 
Choices in this area may be a matter of indiv4|iual style, as much as 
•company policy:, a popular average^ value is from lO-AOX' of total professional 
time allowed to acquire and pt'ocess relevant information. This may vary 
from zero , to 50Z or more at times, but the manager who values the Infbrma- ' 
tion approach will allow extra time for such work, where the manager Wh6 is 
unsympathetic may not. Most of the information scientists interySJwed in 
this study agree that the, basic time allowance which the manager sets for 
thfe people under his direction is changed very little by the success of a 
new technique, A few are more optimistic, but there is a strong* don&ensus 
that the time devoted to more effective Information services is essentlallx 
subtracted ftom less effective sources of information; total time allowed/ 
will remain the same, 

• * 

. The choice between .individual research vs. team efforts is a 
matter of company policy, as yell ^s the managers involved. Information 
analysis tends to flourish in a team environment, and it may be less 
appreciated where individuality is strongly emphasized or completely denied. 
The emphasis on individuality can be overdone either way, with complete 
isolation between workers or none allowed, and the Interaction of selection 
factors in this area merits. further study. 

^ j: 

The lmi>ortance*of innovations also depends^ cm whether jnanagement 
does or does not want anything new. A new develppmetft may be a threat to 
business, of direct interest' in a competitive situation. Or it may be^ 
a threat to established practice in some^specif It aspect of the business 
where a change is deemed inadvisable for any reason. In *either of these'' 
cases, the information desired may place heavy emphasis on finding reasons 
why the, new development will not 'work as described. The ^ormal assumption 
is that innovations are desired, but that such information ^l^mostly f or 
current awareness jand much less often for specific action. 

The question of whether there is^r is not a premium on. the 
quality of_ information supplied for a^^given request depends partly on the 
specific project! but it oftfen invdlves matters of company policy as well m 
as the manager's style. Whole arpas of casual' interest may be defined in • 
which only superficial information is desired, with* no premium for details, . 
and other areas where specific details are of interest byt no action is 
planned barring some .major surprise. These serve as exclusion parameters, 
to narrow the area' in which it pays to even consider coverage in depth. 
Management policy as to the extent of concern in a whole related' area enters 
into, the decision as to the depth^of search: is the area worth while, 
regardless of the specific project? Alternjitlves may be compared in teihs 
of the risks involved in accepting a quick answet, or no^answer, rather than 
looking for ^ J>est answer to the question involved. 
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The time constraints \Alch apply to any search for Information 
Include several different variables.* Times for performance which are^of 
Interest for research or for operations ^ipay Include the tjLme for soonest 
notice under favorift>le conditions, a€d some measure of the risk of late 
notice when things ^reaH wrong. Both are vital for patent problems, or 
for competitive deyelot>ments of special Interest. They are measured onl5->. 
a pproxlmately^by the average time for perfoilaance, which the vendor of 
services woruld like to use as 4iis yardstlclc; All of these refer to the 
time It takes for the service to process Information from the. current 
literature Into Its finished product. In an abstracting or Indexing 
service, this processing time must Include whatever Indexes or other ' 
means of access are normally supplied and required foi^ effective- use in 
retrieval, ' as well as for current awareness^ / 



V 



.The time constfalnt for management includes time to let the 
information "soak," to rebaletnce the rational and subjective and judge- 
mental inputs in reaching a final decision. Similar parameters of time ' ■ 
apply to the problem of th^ retrieval and use of originals which are 
difficult to find, as noted aboVe, for u^ers who require the original 
to meet their needs. The extent to which these requirements are met ^ 
enters into tlfe reputation of a servi'&e for future purchases. 

The tolerance allpwed for extra time to reprocess information 
as received before it can be used is another variable. The concern for 
such extra time can be a majoi;^ threat , when there was a need or expecta- 
tion for direct use-. The supply of more information, even ±i it is better 
information, only makes the- pro blej i worse. if it still' has to b^ digested 
and theiif is no time available.* , ' ' ' \ 

THe question of timeliness has quite different meanings t^o 
different managers and users: appropriateness , for unscl^eduled need, 
dependajblllty according to schedule, or immediate response to urgent 
reo^estw These can be linked directly to the "function of usfe (for 
T^earch, operations, or planning). ' Similar conflicts in meanings and 
Implications- are found for many par^eters in the selection process. 
This is discussed further below (see Table 6).^ 



Thfe maturity of the specific project interacts with both the user 
and with management concern^ in deciding what type of search is desired. 
It enters directly into such questions as ^to how essential It is to ^et how 
much ^information; and when. Breadth of coverage, ^detail,* and when to stop 
are ^J?irt;eracrive decisions which apply at all stages of utgency and justifi- 
able expense. * « 

.Types o£ search which may be required at different times are 
stimmari^ed in Table 5. Th^ arrangement is essentially chronological,' and 
in Increasing order of' urgency. This ta|)le was derived from variables 
Identified in different" types of patent searching and generalized for 
other areas that are now equally Important*^ Patent searching la one of 
the earliest areas where diffeneifS^s in procedure were systematically 
developed for different t;^es of information need. 
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Table 5 ' 

. V — : ^ ' 

Types of Search at Dif^rent Stages 



Tyi^ of gearch 



Project Stage 



CharaQgerlstlc Concerns ^ 



State o£ the Art project planning, 
, ' . background 



Novelty 



Cd^tracts 

( . 

t lability 



• for research,' patents, 



btoad view, moderate depth; 



markretlng, or tnanufacture' what has been 'done and hov a. 



specific data 
positi"ve or 



a, ^ du^ing research, patents, 

negativ? / m^rket>ing, manufdol 



define limitations, jor negotiations, 

ranges, alternatives /infringement study 



civil limits, cavekts, ^ / ^dll of* the above 
goveriiment regulations / (FDA, EPA, etc.) 



what worfl^ifcRnd do^sn^t; methods, 
what will appeal to the%ublic 



what will other managers pay for 



evalulitions^ differe^es, 

best technblbgy-; ultimate effects 



no 

I 



\ 



^0 
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^Jfbroad state-of-the-art review before prdj)Bct planning may be 
, requested^ background fqr'researcji, patent pianniiig, marketing, or 
manufacturing.^ The novelty search is more specific: it is needed dhce ' 
a project has been defined, to determine whtft direction is best tor the 
new development. This also applies to the' complete spectrum of use, for 
research strategy, p^tejit drafting, market develppment, ^or methods of . 
. manufacture./ While contract negotiations often center on, patent infringe- 
ment, they may be generalized to define and clarify those 'aspects of new 
development that someone is wiling" to pay for. The overriding question 
of liability has been broadened far beyond^ the caveats of business law" 
i>eeausfe of the goveriMnent regulations issued by a large and growing list 
,of ^agencie^. The Fogd and Drug Administration and the Environmental 
Protection Agency are typical. This effect is discussed further in Appendix 
\ ^* large^ requirements whidh these agencies set for coiftplete 

. evaluation,- bes^t available technology, and environmental Impact statements 
represent a significant part of the present total demand for special inRr- 
mation\ services. 

A major information searclLinay include any o\ all of the f unc^ns 
.of research;* operations, and pl4nning, toprovide backgrouiid for project 
definitions, evaluation, patenting, publication, or propose^ to management. 
Any of ^he^,e types of .search can be broad or narrow, simple o^ complex, 
short or; long; * DeptH of search and the decision when to stop are parts ' 
of a dynamic process. In general, vou stop when you get a good answer. 
In reference calls, to retrieve ^ sp^ifie document or a known fact, the 
answer is obvious. l?he. less defined i^your needs, the more expensive is . 
the, service. In a patent novelty s^arch the right answer may be just as 
obvious, but either simple or, f^ more difficult to obtain. Breadth of 
search and the nun* er of specirfl services employed can be expanded 
Indefinitely 'In many such -cases, depending* on the skill of the searcher 
and the Importance of the subject. The customer (attorney) in su6h a 
patent search may do all of the analysis of inferences supplied and make 
, all thes decisions fdr search, or he-ma^CTiploy the services erf an informa- 
^ tion analyst who can conduct aqd modifjT the search more or Oafr completely 
on his own. The more cpmplaiMhe search the more continuous A" the inter- 
, action required, whatever t^e of use or service Is involjred, aad the more 
'important it fs for the service to be able to accomodate' ind £f possible 
anticipate ^hese changes in 'depth and dir(^tion. 

Conflict in Meanings of Key W ords 

The semantic confusion of confljlctlng connotations for the same - 
,key wor.d8 can be %duced by associating sets of meanings with each of the 
functions identified. This is an end product of working out definitions 
which are acceptable to people who 'disagree. Conversely, these different 
set;^ of connotations help to characterize the functions with which they 
t'^iL v''^®**' JP^^ illustrated ^f&r eight "parameters of value to the user" 
^^JTable 6, irfhich appears in the- Summary (page xvi) with accompanying text. 
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CASE HISIditlES IM SfiLECTlON FACTORS 



The examlfiat ion' of variables in the selectio^ process and linkages 
between them started, with the search for identifiable user populations arid 
dimensions in t^pii;^ revaluation, as discussed above and in Appendix B. 
Surveys and interviews developed as empirical concepts tlie" interaction 
effects of user priorities and the environment of use , linked t^ the function 
for which the service desired. Actual c^ase histories/noted , during thip-- 
^development illustrate typical, situations where a. nui^e^ of variables are 
involved, all at the dame tl^e. As 4 matter of policy ^n this report, 
locations and examples are not , specif ically identified, 

Initial Purchase of an '"Expensive Service % 

The first example referred to an expensive indexing service which 
had been examined on a trial basis at a central library butuef t untducbed 
on the shelves and returned to the vendqr. It was unfamilia\c to ^tentiaX 
users, and required training to bfe useful tb them.. While the Initial^ ^ 
reaction was negative, trade infoinnatfon" over ensuing* years indicated 
unique potential advantages and satisfactory exper±fenqfe elsewhere. Oir 
a subsequent visit* to the vendor tollmp^ed indexing system was offered, 
based op' additional yeara of experiei&e, but r^quii?lng a substantial initial 
purchase cost. By happy coincidence, upon returrtine from this trip, the 
manager^ concerned was infonped by Acbountiriy tHat/dfi' unanticipated balance 
of $X, 000, was left In a capital reserve accoim't-^ jind^the n^w ^s^rvice was 
purchased at once. Six weeks later Accounting called again to say thfey had 
made # mlstakfe*, but by that timd the new eervic^ was**lMady in use, with 
enough' experience to easily just*lfy.its contimied ^cost. 



Analysis of this story reveals feeireral -factors: first, the ^ 
"energy barrier" to the initial purchase w4B |ipparently. too high. Al^hg 
a different line, this and similar -experiences in the *evaludti<Mr of research' 
innovations suggest' that a successful ^ experiment is frequently accompanied 
by what appear to be lucky breaks but are in fact a baajs for serendipity 
or management hunches, based on expprieiicte. The exact converse of this is 
that in experiments which turn, outk to g^ve negative .answers, or/' failures, 
there appear to be many unexpected jgremlins or quirks which^ there was no ^ 
logical reason tb anticipate. TheJJi?^qu€5acy such event Suggested that , 
intuition or the manager's hunch' Js in fact an ^s^^ntial dli^Hfiion to con--* 
sider in the decis^^||^^making procesi. ^ ^ 

^ second ease hlstor^ ?JldetitliEied was somewhat an^ogous. In this 
case a major, service offered' fiK^verai sictlons had been purchased (in 
part) by an Affiliate Ubrary> based on Inf orjaiation in sales brochuresr 
It had piled up unused on thi^ shelves because of the considerable cleri'bal 
\lme required to set it up and get started, .and. to fifterf lie a<faitlonal 
sections as received keep it current. A major search |eque# was 



received' at the res|^h center ^ich could justify the tLne to convert 
this file to usablc^^^orm, and it was boxed up and shipped off promptly ^y 
the afliliate which was glad to clear itr «lielves. The resu^^gain was 
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a complete success, and led to expanded usefulness ba'sed on further 
experience. These cases' confirm as. major variables in the selection, 
p«)cess, the persoimel time required for maintenance and training, as 
well, as initial and continuing service costs. 

Selections for the "Approved Xist" * . * ^ 

The*, normal everyday problem confronting the user is which infor- 
xnatlon service to ^ick from a JUst of those already available, not the ^ 
purcl^ase of a new service. The approved Ust of ^services he find^ at hand 
is based on decisions made at another time and place ^ frequently oj^ce a 
year. While this previous decision is based on contributions from all 
•users of the service, the indlvidiml may be quite willing to defer It 
entirely to someone else. ^ , 

Almost any user 'of inform^l^on Services has access to the services 
of S9me manager whose exppjrlenc^e he can consult, for daily i^'e as well as 
for adiditions to ^he approved list. This is frequently the manager of an 
information center or the managelt of an information serVice. Both of^ these 
are Jvendoi^ of , a product wTiich they must sell to stay in 'business. They 
share a commoi^ interest in prices, production costs^ and quality standards 
which can be set as specific targets. Quality standards cover a very wide 
range:' managers interviewed report that many services have no standards at 
all, except to get the information out. Some of. the simplest quality 
standards are the requirement that evefy item of information supplied must 
come from a known source, or the ability' to train staff so that the same 
* question can be given to different searchers and come up with the same 
, product. ' ^ • * ^ 

Cost coiiq)arison between cptaipeting services is an area of decisions 
most frequAxtly def€?rred by the worker to someone else, .in' administration or 
management. It is a copnon observation that the scientist or professor or 
Supervisor whb decides to Wi^ a service on hand is actively annoyed, by 
being asked think about costs, and if the require^pient is enforced he 
identifies ±^ irith discouragement. 

> \ ^ - * ' \ 

The administrator vJho makes these decisions finds that simple cost 
comparisohs for budgets and/ accounting are tempered to allow for differences 
in the service of f ered.> - The aMiaajytion is that market .forces keep directly 
competing services close to «tfch other in price, and that the seBvice \fh±ch 
costs more 'has something more to* offer - if this feature is one you^wanx to 
buy. .To the extent Uiat this is true, cost becomes' a secondary factor .in . 
the selection between services, as long as they^are withifi the same general 
range. This may be entirely contrary to the trade literature published by 
the vendors, who are highly co^t conscious arid anxious to use' this -point 
as a selling tool 
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On thi^Pasis, there would be only threfe sigfiif leant, ranges of 
cost: on the market, too cheap, and too expensive. Differences within 
each range might be considered as relatively unimportant, although .varying 
over as much as one order of magnitude including ^ average. Frices far 



below the average are suspect, because they suggest dangers to privacy or * 
soBe defect in the sepice offered. Adjtfstments are necessary for the 
number of customers, varying from a seryice which is <|ieap because of cost 
sharing to a service which is proprietary and expensive becatTse any dis- 
tribution to pther customers Is rigidly e^ccluded. Prices within perhaps 
50Z of the average are assumed to correspond to desirable extra features 
ot allowable omissions, and may not be questioned seriously as long as the 
user has ma<}e up his mind as to what he wants. Prices above this range 
Wing -in new questions of -their effect tn total budgets, of available 
alternatives, and of the probability of use sufficient to justify special 
purchase. 

An example df this factoi? of cost by range is the decision of a 
central library to cancel its standing order for continuing index volumes * 
in a series such as Beilstein, which are i^gcognized as valuable tools for 
access to information. The problem is that Beilstein in Supplement IV has 
now reached the range of $400-500 per volume. Five such volumes appearing 
at unscheduled intervals in on# year can ruin the book budget in even a 
fairly large library. Prices so far above the average force the careful 
consideration of alternative sources and ^procedures, no matter how useful 
the item. 

/ 

The question -of probable extent o,| use interacts with cofets 
somewhat differently in different ranges. ^For average coats, even, within 
budget, many libraries insist that a single tiser must make the purchase ^ 
recommended with his own funds, unless he can demonstrate that the service v ' 
will enjoy multiple use by others as well. This requirement is most likely 
to be waived for requests of low or average costs. In a new field of 
interest, where there is a presui!q)tion that others will become users after 
the new seryice is made available . Predictions of probable use and 
acceptance become ,more useful with Increasing skill and experience. 

Special Topic Bulletins • 

Th6 opening of any new field of scientific and technical Information 
is likely to be greeted by one or more new newslAters or special^ bulletin 
services, as soon as it> c^n be expanded into a recognizable market. The^e 
bulletins may be aimed primarily at the planning function for It small user * 
who does nat hav^ his own information service. This -may be an individual 
in even a large corporation, however,^ in a new field where .interest is 
liiftited, or where the user does 'not Iboy^how to ask for information. , Since 
these services are expensive and tentj'to Become full of trivia, special ^. 
quality factors can be developed for their selection including such criteria 
as the following: % 

. - ■ ■ 

qualitative ' ' , 

- itois come ^rom primary sources, not just old material 
JihformatiorNcumulates, access routes supplied 
'tied to khown sources, improve access to other material " • 
suitable for browsing, organized by "related Ideas 
honest claims as stated, fot breadth and depth 



judgmental | 

- repiltatlon of publisher, probably a quality product 
Importance to major company interests % 

quantitative * 

T priced within the current market range. - 
number of readers, sufficient to justify Central purchase 

As- a ''selective reminder" of felevent information for planning , ^ 
use, these servicies are in competition with S.D.I,, the trade literature, 
.and review articles* in the stjan^rd journals. The one characteristic in 
which ther^ was the strongest aPRment 'in' planning staff interviews is 
that a special information service may be of no help unless it is *^ 
provided by an individual whose judgment San be trusted, based on aperience, 

A University Science Library 

A comparison of the. work environment an^l other variables between 
a University and an industrial^esearch laboratory indicates some shifts 
in emphasis in the selection of p,urchased information services, but many 
of the sam^ principles apply. The following list was suggested at a nearby 
University Library of Science and Medicine: 

\ 

- The Library *does no searching as a regular service, 
for itself or for other people, but only provides 
tools for others to use. 

- Semi-public use of the facilities by unskilled us^rs 
creates serious problems of -misf iling, as comparea to 

^ the industrial library which inay enjoy a more protected 
environment and users of higher average skill. Thus^ 
the University Library refuses to purchase card . ^ 
services, or loose-leaf with a steady flow ofv sufiple- 
\ mentis, because th^y^ i'nvol\r»x^block of time to get 
inf^^usiness, interminable ffibsrf iling thereafter, 
and the constant hazard of items which are lost or 
misf lied. • 

- The Library is leary o£ highly specialized materials,*^ 
arid requi'res a minimum number of users, no m^tfer who 
they are. This appliaie particular^ly to special _L 
Interest newsletters, which are used for retrospective • 
searching very seldom, if at all. 

- Any individual' research project orjcontract reports 
can be terminated, ^nd leave an orphaned, broken file. 
An index or accAs system which is needed by only one 

- man must be purchased maintained by him, therefor/, 
out of project funds, regardless of how valuable it is 
to him. Such a file may be transferred to the LiJ^rary 
at abater date, but only after sufficient use by*" 
% others has^^i'^^ established by experience. ^ * • 



- Uniqueness of a giv^n service on, the market compared to 
Its competitors comes before cost, in deciding which 
ones to keep which to discard, although both factors 
dffe 



must be considifed. 
Upgrading the Special BVanch Ubrary 



\ 



The problem of how to improve the operations of a special branch 
or departmental library without losing Jts advantages o^ even destroying 
it goes deeper than personnel ♦ It Involves what the system is and why it 
• exists. The small special librai^r may be in essence an expansion of the 
personal library of the department' manager i created as a service center 
and not just as an^^formation source. It ojSerates in a tlosed market, 
which is not likely to be expanded regardless of any Impwvements. The 
special services it offers serve the department staff as a way of doing 
business, and limitations on the tiflie of its ufiffer/customers are as ^ 
stringent as limitations on* cost* 

I * - 

A detailed analysis o£ ^uch an operation was made for the special 
library serving an environmental health research unit* The first con*"* 

, elusion was that even this small unit covers a complete spectrum of 
demand, in its use of literature servicejs. Toxicologists are doing basic 
research, with heavy use of the published, literature. Industrial hygienistai 

- are operating engineers, vho consult company sources first and find the I 
literature less informative. ManageineAt planning^is concerned with govern- V 
ment regulations and new areas where it must be able to get relevant infor- 
mat ion immediately *on demand. ^ ' 

All three 'groups are users 'o& information services, biit in quite 
-different ways.^ In this particular location, the typical research/toxicologist 
likes to lip his own searching, asks the librarian where to look (and* 
whether it is worth looking), wants *the special search files kerpt up to date, 
and often want^ to get specific references in the original. The industrial 
engin^er/hygiei^ist wants the librarian to provide the'search, seldom does, 
his own, and is usually satisfied with a collection of whatejver references , 
are typical and available. The manager requires the monitoring of specific 
government documents arid company literature, and likes to get on his desk 
with the new item a pre-selected 'Vertical file" or relevant documents and 
reports, /i . * . 

These three groups correspond closely to the* three functions of • 
information ilse outlined herein. To a certain extent th'rie U a tendency 
tor each group to assume that its own part of the operation is most 
important, and take the others f6r granted. Thus, upgrading any one aspect 
of the operation is not a complete answer^ Success may b^dkfire and make 
other users suffer: if an improvement in one function alone makes that ^ 
l)art of the service better and justifies an increased demand, the limited 
time of the staff must be taken away from others, unless improved efficiency 
can be engineered to release time for other functions. 
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In this specific oase, ah> analysis of the special services performed 
by the staff w^s made to dfefine those skilts which are valWd most hi^ly, 
both the staff and by its customers* The primary sfills identified were 
(1) upgrading the input to the system by selecting and re-abAtracting all 
originals of ^elected documents, for hidden d^ta or meanings Aot adequately 
recbgnized in the abstract (or in the original); (2) creatinglpre-selected 
vertical files for major areas of current or expeated interest; and (3) a - 
current awareness and retrospective guide for where to look, for what is 
in the files. . • 

Theise "premiun skills" ire listed in the inverse order of which 
ones might be helped most easily by ii^)roved computer services* • Thus, the 
guide to what is in the files may need only shallow indexing for the'cojnpany ♦ 
literature, and the basic research literature will benefit more quickly 
from commercial services shared with users at other locations. Computer 
services may also be able to assist in items (1) and (2) by providing the 
staff a selected input for special process"ing, but without in any way 
teplacing the total operation. , 

The basid principle in this ca^e, as elsewhere, is to look for 
help *in areas that are weak, so as to preserve strengths and provide mofe^ A 
time to use them to advantage. \ \^ 
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- r 5. CONCLUSIONS 



The Matrix of variables and interactions proposed suggests ' 
a number of relatjmiships %^lch have been stated above. Seme of these 
have both positive and negative elanents. The positive statonents can 
be summarized first: 

- Use as aii .event is much easier to^escribe than the 

• user as^an Individual. The satfiel^divldual may be a 
research scientist today, a searcher for a known 
document tomorrow, and a project evaluator next week. 
His nee^s and choice of services to f Hl.them^hange 
directly for each f un<;|:lon' or role. 

" The concept of user value-systems' is an extension of 
the identification of user-groups by function. Users 
concerned wl^h the research functiqci appear to place 
• . ' the highest value on qualitative factors, the opera- 

ting function emphasizes; quantitative, fa<!tor^ (cost/ ^ 
budgets) and the planning function the facCbrs of 
management judgment. ' « 

, - The f^ct that an item of information is new to the user may 
be enough for some (R/D planning, marketing), but for others 
users evaluation is vital (patent^, contract negotiations). 

^ - Cost is a strong factor in deciding whether to purchase 

services or not, but once this decision is made, relative 
costs of the same order of magnitude may be only secondary 
in the cholqe between services. 

- Information analysis differs by def Inftlon from a search 
routine which "anyone with adequate training can carry 
out and get exactly the same anser." Th# value judgment 
^UdLskllls of a specific individual are a necessary 
part of the product. . - ^ 

Negative Conclusions or Unllnkages Observed 

Many conclusions can be stated more cogently in negative terms, 
as unrllnka|es" or falsft correlations to be avoided: 
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No matter how thoroughly well developed any one dlm^slon 

may be in the user-selection process, it runs into- trie ^ 

lav of dlmlnshlng returns. 'What is most needed is 

whatever is missing - a bit of public relations or common 

sense, ^wT^coat data, as another valid way to look at 

matters from an entirely different angle. The person 

least sensitive to values in another dimei^sion' may be 

the enthusiast who thinks his syatem could become all ' 

things to all people if it could only get «AOugh Wpport!. 

A forced decision between services based on direct dollar 

costs may be a bad decision if it has to be made by a 

user who prefers to think in terms, of quality an4 is not ^ 

trained in cost/benefit analysis. Costs are important 

for the oper'^ting manager, but most users put convenience 

and reliability first and consider costs only within an 

order of magnitude. 

There are negative values, not zero value, for information ^ 
which may be at-tractively packaged but never used. There ' 
is a recurring debit each time this >item is considered and 
rejected again; it^ takes up space and time^ and the system 
would be better off without it. Thts minus value can be 
projected in mathematical terms. as a vector transform, from 
an oi^JLgin in wishful thinking to a revised basepoint of 
more realistic neeBs\ - 

There are also strongly negative values for an '^important 
reference" for whith nd original can be found, for Informa- 
tion received tqo iVite, or for any information which is 
rejected^ for cauge and left in the file without annotation, 
to be either wrongly^accepted or reprocessed for another 
xejecti(^ ,the nej^t time around. 

Time involves major factors in indirect costs which .are i 
of ten /overlooked, particularly the time required by the 
user /custome r to rjeprocess information which cannot be 
used directZyas received (e.g., a 3-inch stack of com|>uter 
printouts)( 
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APPENDIX A f 
PROJECT APPROACH 



Blas^As a Working Tool 



ltrT.8 an interesting exercise to read backwards through the trap- ^ 
script of ^ panfel discussion on Information systems and services, and ^ 
discover how easy it is to identify , (without labels) who is saying JwhafT 
Differences of this type appear during the evaluation, of any lai*ge body of 
Information, ijivt?lvlng the value judgments of different men fr6m dif- 
ferent points o| vlfw. How to handle subjective differences in' these . 
points of view ^n be a serious problem, that is not, always recognized. 
One approach to^hls was developed in a project on the "Evaluation of 
Sovlet^esearch in Catalysis, V completed at Exxon Research and Engineering 
in 1^0 (15). The problem in^this instance was that the ratings of the 
experts in differ^ parts of'^the project werfe. difficult to handle, since 
th^y were morept less heavily biased; many of them said the Soviet results 
were no good lit all. - v- 

In this case, some 6,odb pertinent abstracts were collected from 
the Soviet literature and rough-cut Injo a dozen subject areas. Each 
subject, area was assigned to a cognidtot;. expert, .who selected from his set 
(or reclassified to other sets) those Items which he felt to be most signi- 
ficant fot further study. These items, representing about a lOZ sample -of 
the whole, were then crltlcaily evaluated by each expert for his own area. 

The ijlltlngs of these experts were then collated for further analysis. 
At tMs point it feecame» apparent that, in- many cases the same project or 
program had been rated quite dlf ferentl^^f b:^ different experts, each from his 
own viewpoint, and these ratings werj^ not .?i#all coiiq>atlble. The next step 
was to consider as a group .all of the ratli?!^ of a given expert. When this 
was done it appeared that one expert might feel that In his area all of the 
Soviet research suffered from fa setlous lack of experimental data. For him, 
.the statement that a given report contained somei poor experimental data might 
turn out to be the highest compliment he offered. An effort was made to 
determine the bias used by each expert,, and his rankings were then arranged 
in the order of what ^eemed to lookrthe beat to him, within the Constraints 
of his own vocabulary. On this basis there was a much closer, agreement ^ 
between the experts. 

► *^ , ^ 

The next step was tor Identlfr afid then* compare the differences In 
•the bias of different experts, to look for a second geperatlon of cp^iclusions. 
One of. these has already been mentioned, in the apparent' dlfferenrfP in the 
attitu^of Soviet^ and U.S. scientists toward the Importance t)f expftrlmental 
data. This derivative conclusion could be checked independehtly ag|^8t the 
observation that^some of the best-known Soviet scientists in catalysis 
(e.g. Balandln) seemed to he quite erratic in the quality j^f the data they 
reported.*^ It appeared on further study that tJie q;ualltyW tjie data in these 
Instances could be correlated well with the later quality of the' work done 
by the junior author, but not that of the senior author at all. 
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This approach Is similar to tjhe perturbation analysis .of astro- 
nomical 4at4 to JElnd an unknown planet. An dnoTmous mass of data Is first 
arranged to determine that some bodies seem to moVe In orbits among the 
stiiiff^ On closer stud^lt appears that these orbits show some erratic 
effects, and areas ar^ldMtlfled In which these effects see^ most 'likely 
to occur. Theste effects are theft examinee^' from all angles to see If a 
*rea^9n for them can be surmised. A variety of teijtatlve conclusions ^ife 
.made and tested against other known facts, ^nd the location of a new 
planet involve^ a third or fpurth (derivative ot the original orbit..: 

The perturbation-analysis approach examines w^io is saying what, 
and why, and goes beybnd this to consider the reasoAs for the bias of ' 
different experts. It is a direct opposite in this sense of the Delphi 
approach, which masks differences in anonymity in order to arrive at a 
conseosusr. In the Soviet ^tudy, pertinent conclusions directed attention 
to the. lack of training in Soviet universities. in what we call chemical 
engineering, but with the companion observation that work of the highest 
quality can result in areas of R/D that receive a high "natioi^l priority." 
Specific conclusions directed attention to at^as which would be\of "probable 
interest to the i?SSR .(as of 1970) in any future interchange wltl^^the U.S. 
in science ax^ tecrinology^ < " 

These prlnclp^ff w6r6 derived from, the analysis of subjective 
observations.^ They have since been tested and in general confirmed b^ /'^ 
subsequent experience. The basic technique is to select ar^ cyf in-. ; 
formation for careful scrutiny, based on preliminary correlations, . to ^cu 
predict the nature of a variable and \^here to look for it. This correla- ^ 
tlon can then be checked against what may be very fragmentary data at a 
specific point, for ftfrther refinement. 

-The application of the perturbation technique to the present study ^ 
Involved an aggressive search for areas of agreement and disagreement among 
the experts interviewed. The object was to detect and define significant 
variables in their criteria forr the selection of an information service. 
As this approach was confirmed, it became standard practice to *(1) ask 6ach 
expert to identify others whom he would expect to dlsAgree with his state- 
ment 61 priorities, then (2) go* to these or others like them, and (3) work 
out between the two a statement of variables and deflriitlons ac^ptable to 
^both, to describe the differences In their points of view. The emphasis 
on examining reasons for individual value systems requires a continual - 
awarene'^s of the personal bias of the investigator and the special environ- 
ment of the study. The awareness of these hazards helps to avoid their 
effects, and 'It is also helpful in interviews continually to seek for aieas 
of disagteement or divergent viewpoints. .'^ i 

' • • * \ 

^ The special environment of mucl]^ of thdrs study is a corporate , 

research laboratory where In-housb infornqkation serv ices are highly valued, 
and are provided as ^ corporate^service function to^ll pr2fess^onal 
eiQployees. The effects of this bias are recognized herein as an explaT^atlon 
for some of th'e .statistics obtained, and also for differences from 'selected 
interviews with th^ managers of a variety of information service operations 
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at other locations. An example of. personal bias 1? the tfendency ta drjaw 
generalizations, which must he guarded. against as a hazatd of ambiguity 
i4 the results> ^ hidden constraints. For this J reason, the conclusions 
*drawn herein are concepts to be considered as a skeleton for the organiza- 
^tion o£] ideas\— aa a guide rather than a directory. f cpy 

Dimensioggl AoaLysis , ^ ' 

Business literature on the 'evaiuatj.on of research gbes back for 
several decades. Efforts to relate the value of research to ordinary " L 
bus inessr parameters alone (such as DC?) have dhown that this approach 
* sacrifices entirely too much regardingr the value or probability of a 
different outcome. Improved systems for the analysis of uncertainties f^nd , 
some means to^include anrf to operate on 'subjective evaluations. In a one- 
dimensional analysis,, uncertainty factors an^ numerical factors such as cost/ 
benefit aspects are frequently entwined, and it is #iecessary to resolve them. 

The dimensional approach to' the analysis of complex ^variables 
in project evaluation was advanced 15 years ago for . the rational selection 
between projects in industrial research. Harris (Monsanto) described ^ 
profile scheme with 5 "aspect?' as headings for 26 factors, each ranked 
on a four-unit* scale from very good to very bad (16). This system for' 
the evaluation of research was not widely adopted, a^d* later studies 
----^^ggested that 5- major headings is still too cumbersome. A more limited 
tuhnber of dimension* is preferred (17). Three^ dimensions has this 
advantage and is the* minimum required to escape' the hazard of "Flatland": 
false cause and effect conclusions from two-dimensional thinking regularly - 
assume that if a thing/lLs not black, it must- be white, forgetting all 
about ^ed or green (l8). . ^ ✓ . * 

System's of this general type have' been tested extensively for us^ 
in research planning and the evaluation of projects. One key to success 
to be able to break the analysis down into a small number of discrete 
chunks that are analyzable segmenta that can be examined separately. One 
. such system. to rate the "Value of an idea" in R/6 cliose the three, headings 

^ of novelty, potential value, and abil'ity to commercialize, with suitable 

J subh^ulings under each such as patentability tecbnical uniqueness, or com- 
mercial novelty^. Ratings for each factor were' a subjective ranking of 
value judgments, in terms of probability or extent. Each main«heading was 
then rated as an average of- the subheadings ratetf- separately! A number. of 

. in-house research projects were rated in this way, and wer6 CQpp'ared vlth^ 
a numerical rating of the final stage. reached in their development toward 
a comnjercial success. A plot * of ' thesa data, in Figure A-l» sho,ws a cle&r 
correlation between Idea Value and the development stagei attained. It ^is^ 
significant that parallel rat'ings by independent panels of experts who' 

■ were equally qualified showed^heck results which were remarkably close^ • 
within a few percent- on a nuiBre^;lcal, average of ratings. The extent of' * - 
this agreement was reported with some ^urprise, as it had been by Harris 
and others beiore, since d.t was assumea tfhat subjective ratings wQuld be 
inherently less consistent than those based on physical data; 
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. The dimensional approach makes no effort to compile exhaustive 
check lists pf variables , so the sub-headings in' the tfext Tables I, 2 * and 3 
above are all typical, -no^t comp^lete. vThe principle is' that the same amount 

- Of effort Is mo,re productive wl)en it is spent in looking at things from aq 
. many vieypoinljs a» are necessary to establish perspective. It is the dif-' ^ 

ferences in die^M|ewpoint8 which are represented in the sumlffary Matrix of 
^ variables 'and«teraction effects. " ' 

• TW particular dimension^ chosen in this study to reflect, the 
value -systems of different ua^fc^ ^receiVed a direct inpiit from behavioral 
. analysis. They are taken dire(^fy from- the Jungian system of psychology, 
which considers the objective/rational^ suMecfive/qualitative and judg- * 
mAital/intuitive aspects and their interactions with the environment as 
. fund^ental characteristics of behavior'. These patterns o& behavior help 
^^xplain why the scient;ist or engineer is so baffled when management says^ ■ 
"No'N;o a well-planned project, and , equally why the rational planner can 
expect his system to be aH. things »t:o all people if only, he can get them 
to absorb the necessary cost#i ' . 

. Overlapping New Technologies " , • , 

. V • ■ ■ ^ ' 

. r _gF^^ simultaneous development of different new methods of access 
. to InfortStion confronts the oser ^ith an uncomfortable choice M to which' 
tpols^to buy, and which *utl^^$<^ push. It overlaps significant user- 
desi|||d imp^vements in- the, older f^iliar systems, some of which the 
user TTlmselfJaify have helped create. y • u 

- , . .^k/ Science amj t0chn|)logy hafe now reached the 4th. or 5 th generatJl 
of iji^^Knatioij access tooli| 'with new tools coming on so rapidly that th" 

^ lines, between gfenerations ite blurred. A host o^special services ol every 
tmagl^i^le type are puperimposed on journals," abstract journals, indexes 
to abstracts, and computer access to indexes. The design of new systems 
and services is Itself a^well-established discipline, po that' the .avail- 
4K f^it^ °^ technology is leas and less of a*Iimit. The number *of 
^iipfWmatioA -serV^Lcee available 'is in the tliousahds and still growing, and 
~~6ach-of them has been created to meet th^ challenge of Specific user needs. ' 

, . • Currjp fftdtistics on the groWth of information services amply ' * 

??^^^^'^^1f^L??^''f '^^^^'^ "abstracting and Indexing services" 

• ^/^''^^N^ International Perlodrcals Directory has gro»^.to about 

- ^O-in 1975 froA.12D0 in 197|rup from an estimate of 300 in 195h%mm 
ig mnber of .'.information service centers" listed in Kruzas (International) 
t/.s Encyclopedia of Inforiation Systems and Services rose to 1?50 in 1974 
> (2nd Edition),, an increase cJf over lOOZ nince 1971. The number of computer 
' ?:o?f available for scientific information was about- 88 as of mid-year 

1976, 57 in 1975, and 30 in 1974, .doubling ever^ 1 to 2 years a9). "The 
same trtends appeared earlier in the NSF "Guide, to Non-conventional 
^icitatific and Technical Information Systems in Current Use" from 1958 
^_to 19^6, when the number of mechanical sorting systems in use showed a 
fT-^ubllng period of four years (e.g., edge punched, tabulating" carda, 

peek-a-bpo systems) . T^iis rapid growth was all but obliterated by computer ^ 
■ T^T^^Li'''''*""!'^ *' * ^"^t^i'^ed doubling rate of less than two years, ^ 
from 17 in 19^ to 118 in 1966 (out of 175 non-conventional systems rejortej)^ 



The cumulative effect of these 8tatl3tlC8 Is Indicated In^Table 
A-1. The' rapid growth of the' scientific literature, abstract joum^s and 
more sophisticated electronic methods of access to Inf ifa-y^lon ^as cited 
as -a classic example of the "Diseases of Science" In a remarkably predictive 
essay In 1961 by deSolla Price* The growth of technology and science leads 
to new technologies, and each new ge^neratlbn in rf' glvep fl^ld of knowledge 
tends to grow faster (20). 



Price discusses growth curv^es of three types. *The normal growth 
pattern in ^nature is the.S-shaped symmetrlcail logistic cuifve, wh^e the 
supply of food or "some other re^source is limiting. growth Ases 

rapldit^y at first, has a nearly linear middle i>ortio^Br a few dqub^ng 
periods in time, and then levels off '''just as rapldlflR ft runs into a 
shortage of food, water, or some other basic resource. , The usual end 
result unless theve is some new supply is £hat the whole system collapses. 
This applies to the normal growth of bacteria in a culture, fruit flies in 
a b»ttlfe, or, jack-rabbits in Australia. * ^ 

^ The' growth pattern^for the number of . scientific journals has been 

strikingly' different, with a- doubling perlod^of fifteen years which has been 
nearly conSitant for 16 doublings over 250 years, frtpm the first few journals 
to a hundred thousand.^ This t'a a straight line sem^-log "plot. Price finds 
thei^ame pattern fot the growth of abstract journals which started when the 
nunXer of primary journals exceeded 300, or more .than one man could read. 

The semi-log plot of the gro\fth of scientlMfe journals and abstracts 
has too oft^n been ^^ted out of coittext, lgn4v^g Prlce^s main points 
Continued linear* grow\h ciay ap^ly to money at compound Interest but even 
there it coljLapses if ^he bank falls, and it ceases if the^ Interest is with- 
drawn. An unlimited supply ^of all jie cess ary resources must include time, - 
money, and social v£^ue^i|;as well as foofl or air water or new information. 
Free access to all of these is aar' illusion:. . a constant^llnear doubll^ 
rate for the total numb^^r of» universities ot scientific journals or 1i|||§e;;- 
matlon setvlces c4n contlntie "indefinitely" for only so long as society is • 
willing or able to^ aTlow unlimited aooess to fime^ money and all of the 
facilities required. Tke difference between the linear and logistic curves 
is very well Illustrated by the- difference between theoretical and actual, 
returns for a chain letter, as it runs out of gullible victims. If there 
is ho means by which supplies can 'be replenished growth goes negative and 
the system dle^. The fallacy of the- chain letter applies ultimately to 
^ery situation which assume^ contiiv^Bd ex^nential growth* • 

y Even more instrufitive is the fact that the grawth of new technology 

In a given field leads to families of gro%i^h curves which grdw at' an ever 
increasing rate. The reason for this ±b that eac^ related new technology 
benefits f rom^ the tools ''and .exper|.enc§ of the ones before It^. Price 
illustrated this with the growth rate of the total number df scientists 
which cioubled in the Uaited Spates over the past 2QP years abopt once every 
t6n years, cpmpai^d to^a 300-year average^ doubling period in Western Europe 
of 15 years. »fthe doubling time in Russia alnce . 1918. has been more rapid, 
about once eve/y 7 years, and in Chln^ from 1940 t;o v60 it was even fastiy:, 

^doubling every 5 years. He applies €ixactly the same ptdnclple to the growth 
of new technologies for access to scleritlflllK information. 



Table A-1 



. 4 Generations of Ii^fonnatlp'n Processing 



Information Process 
or Service . - 





Doubling Time 



Access Tools 



Scientific journals 
Abstracts and Indexes 
Mechanized indexes 



Electronic sorting, 
computer 4 data bases ^ 

Information Service Centers 

Searching and retrieval^ 
(with or withont analysis)' 



Information Provided 

Original data 
Ref ex?ence if ^content 
;4adex to abstracts 



Index to references; 
Interactive searching 



Any of the above 



rears 



(Period) 



Current Connnent 



15* • (1750-1970) 



12* 



1.6 
1.2 



(1875-1975) 
(1958-1966) 



(1958-1966) 
(1970-1976) 



(1971-1974) 



apparently slowing down' 
(contrqver s ial ) * 

1950-75 
close to 187?t-.1950 

growth overwhelmed by 
computers 

data bases in 70 *s * 
computer systems in,60*s 



services and centers 
total 'over 3000 (1976) 



I ^ ^ ^ . \ 

* Constant^ doubling times assume a vaitishlng border line between science plu^* technology and other 
bodies of Information. , , ^ 
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-Xhe chart j^hlch Price uses to' illustrate the effect of increasing 
growth rates in such afamily of technologies. has been'modif led slightly in 

\ Figure Ai-2, t^ fit the data given in Table A-l.',The only difference is 
that in his family of curves on, the number of scientists it was tHei-4th 
curve for China where growth was truncated by a cultural revolution. In 

• the growth of information services it is the intermediate curve for mechanized 
index systems vrtiere, growth was overwhelmed by a younger generation. The 
computer systems which Price referred to as "electronic sorting" in 1961 
doubled In less than 2 years during the '60s, an^ this siame rate of growth 
is continuing intp the"'-70's in the phenomenal infcrease in the number of data 
bases availailSV-. The growth of information service centers fits the same 
family of curves. It uses all the tools available, and its doub]^ rate 
of 3 yeai-s ia^ exceeded only by that of the computer data bases, newest 
generat;ion. ^^sL^ 

The doubling period for the number of abstracts and indexing 
services is only slightly shorter than for scientific "journals,- and Price 
drew them as of 1950 ^ parallel lines on his semi-log plot. His plot is 
reproduced in Figure #-3 with the addition of the recent data as a double 
point at 1200/1250. for th#':gears 1973/75. ^This extrapolates as a straight 
line continuing 'the slope of the original data for 1875 to 1940. The first 
half doien -entries /or abstract journals appear off -the curve as precursors, 
.for the same reasons as^ifed by Price for the first few scientific journals. 
They are properly omit1« in determining the statistical ^lope of tlie liJfe. ' 

^° <iata are available as to whether the 'number of scientific 

journals as of 1975 is anywhere near the 400,000 yhich would be predicted by 
a cfRTinued linear extrapolation.* The question is considered controversial, 
and an answer is beyond the scope of this report./ Whatever overall data one 
accepts, there is cleat evidence that by 1960'to 1974 the rate of growth. in 
titeJUnited States vai significantly below that for the rest of the world' (21>. 
The question of what is a "scientific journal" becomes moire and. mor^ dif- 
ficult to define as the borders of science expatld farther into society. It 
might be agreed, for. example, ' that Popular Science .Magazine Is adjournal ■ r 
of technology, i,f not of science. It is not so easy to deci<de wheAer to 
count either popular magazines br more serious journals devoted to ^ara^ ' 
psychology, astrolx)gy, or the mysteries of the pjfra^ilds. «it is no easier to 
decide just where to stop counting the abstract journals. Such ambiguities- 
cannot be avoided if the extrapolation is ccfhtinued toward the point where 
there is one "scientific journal", or more per capita. Other results w&ch' 
Price predicted have become surprisingly real: the costs of each new iechnology 
tend to increase as the square or the cube of the number *f men involved; 
the printed ;p^er may be doomed, unless we can get even better methoda+-«ot * 

abstracting the. abstracts; " and th^most persistent need majf be a nationat"* 
science .polict to set consistent goals, for the effective use of techjiital 
information. The gradual disappearance of printed papers and the* emergence 
of electronic data centers for every man, woman and child do not seem nearly 
sa remote now as thfey did in 1961. The final prediction is that whatever new 
technology comes nextf, it qan only be expected ;to grow, faster, and have a still ' 
shorter time before it has used up its share of all the resources society will 
allow fc^r the growth of, science. ' * ' ' 
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What Will Be Actually Used ? ^ s - 

The designers, and vendors of information services would like to 
assume" that major Improvements In the, services they offer would find new 
customers, jfeit the to^al market expands* only slowly at best, and coApared 
to^ the rapid growth of new services it can be consld^ved as almost a con- 
stant. . To tfa^s extent, any Increase in the aoaounts of tlm^e and money which, 
the users of ^TI devdte to purchased information services will be taken 
away from other ^unctionii, such as laboratory research and development. 
For one service to expaxxd it must take business away from others, and the 
vendor finds himself competing with his own established lines for the same 
customers. ' ^ 

Intense competition places the emphasis on what service is actually 
used, nqt what look* useful. Even the best of setvic*e8 can go out ot busi- 
ness under adverse economic conditions, as happened to Chemisches Ze«ralblatt 
and British Chemical Abstracts In postwar Germany and England. A mdte common 
occurrence is for a new newsletter or special journal to fail to find enough 
customers to survive, regardless of high quality In the first few issues. * 
Technical societ]^ journals benefit 'from having a^ guaranteed market, and . . 
jjenerally improved comoMnlcations In both directions. Thus, a close response 
to *^the needs and wishes^ of-lts customers has b een a^i Mor factor In the 
success of the Chemical Abstracts Service, created 5y ct^ists^fcr chemists. 

The problem of otiginaf sojar^e*8oeum^ts which £tre not available for 
aT)stracts in print may be partly, a question of who wantai ^at. Data from ^' ' 
brief statistical review ba^ed on the Chemlcdl Abstracts Service Source 
Index for the years 1973-74^ Indicat^ that this occujrs often enough to be a 
problem, with sources cited from behind the Iron Curtain. Larson has stated 
that 7Z of all the^ sources indexed are not ayailable in the U.S. (22). Many 
of these are sources cited no more t^m once a year. A random sampling and • * 
identification was made of the sourc^Bllted on 50 pages out of the 2000 



in CASSI. , For this sample, one fourth gf the sources '*not avall<a]ble in the 
U.S." could.be found in other libtaries listed, but thrfee foufths of them 
are not available anywhere* In the fr^e world. \ 

• » at 

A further analysis of this sample is be^-.ng diade with the coopera- 
tion of Chemical Abstracts Service. One assumptl^ which can be'^Qiacle 
depends upon the fact that the professional seafcheiis who pre most likely 
to talk to CAS are often willing to accept references in their reports by 
title onTy, or title plus abstract. The scientist or other researcher who 
wants the original document is a step removed from this direct cJbact. 
When he faAs to get what he needs, his sense, of frustration is directed 
primarily ajTthe librarian and may stop there. Such a reference is of 
negative va^ue to the customer, not zero value, since it wastes more time 
and' effort ewry time a potential user finds it , interesting. One approach 
has been to question the practice of even having an^bstract which may 
have been translated from Referativnyi Zhtimal, when it proyes Impossible 
to get morp .information. A simpler recommendation to save this annoyance 
and expense would be to place some definite marking in, the abstract or 
index whenever it is known that the ^iglital is not available. This cou\d • 
provide the citation for what it is wbrth, without the frustration of 
trying to find more> . ^ * , . 
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quantity facitor la the dem^^d for information services hjis 
probably been distorted In recent years by subsidised government requests, 
for "all the Information^ available" on a given mibject. The effect of^ 
these open-ended r^questd Is hard to place in p<roper petspectlve'. 
Govenment agencies had the first computerized Index systems, In such 
tplaces as the U.S. Patent Office, NASA, a^d the Bepartnfent of Defense. 
Civilian agencies were Invited to bid for government contracts to assist 
In ev^uatlng the Information available Id these files. Many silbsldized 
.contracts of thl^ type^ were let by the Environmental Protection Agetcy and 
other branches of government. Their purpose was to evaluate the Informal- 
tlon collected for publications, on criteria^ fop standard ||ettlng, or the 
beat control techniques aCvallable for potential pollutants In *alr or water. 
Other subsidized requests* were simply bibliographic at'^ first, to load« 
potential source material Into the files. • < 

■ ' . • ^ . ' >" . • 

As the number of these subsidized requests increased, more users 
became involved^' and the information' tools ava'llable for them were corf* 
tlnually improved. The literature review for a 1 to 2 year project might 
be^ completed in two to six weeks, with a large number of' references but 
no extra time for their evaluation. Direct subsidfes fox simple biblio^ > 
graphic compilations have t)ecome unusual, and the same thing wiir prje-^ 
^sumably happen to the demand for published con^enditms of all the infor- 
mation available on a special topic. The distortioi^xomes about when it 
beconjes progressively easie¥ to produce references in any quantity desired, 
without requiring that they 'be dige.sted or ueecP. ^ 



The qijiality factor in the growth* of services into new^ market 
areas h^s a different set of ^limits. These can be illustrated by referring 
to Prices' chart on families of new technology, in Appendix Figure A-2. ^ 
The competition between overlapping new technoldgies means that most of 
the market for the youngest" genera^tibn^wlU ordinarily be subtracted from 
the bu^i^ess avjallable to its p^edecessiors, A rapid new develooment may 
break through the previous limits to growth into a new market^^^^ above 
the earlie^r curve, only when it can dr^w upon some new resource not pre- 
yioully available. One example of such a new resource is the high speed of 
computerized searching, coupled wfth multiple uses for the same user train- 
ing ^nd facilities. The trouble is that t^is may come at the expense of 
other parts of scientific research, not in addition to it. It is too soon 
to tell whether the enormous amounts of information poured into official 



rements for environmental tmpttct statements or best available technology 
o the total amount of time ana money devoted to science and technology. 



e moment it appears' 'that until they are completed, they are displacing 
aspects oA^research and development. 

Th^n-nte^ractionlTof Aew technology and its potential users' can 
be verydiMppointing. Two cogent statements of Ijasic principle in this 
area^tdve been made which are entirely negative: ' 

Mooer's Law: an Information retrievA system will tend not 
to %e used whenever it is more painful and troublesome for 
. ^ • a customer to have information than for him not to have 

it (23).^ 



Zlpf? La4T as modified by. Allen: statistical andlysis of 
<^ the choice of information channels shows a direct relation- 

ship to earse of access, with far less attention to the 
effective use of time, and no correlation between perceived* 
technical quality and frequency of use (13). 
%•',.• i 

, Mooer s Ldw can be considered as a modern paraphrase of the old ' 
maxim Where ignorance ia bliss, 'tis folly to be wise," and both of these 
statements are related to Zfpf's original studies on "The Principle of 
Least Effort" (24,25). Allen's study of the information source-selecting 
criteria of electrical engineers at MIT is being updated in a new edition, 
which reaches the same basic conclusions. Others have. reached the inference 
that the ease of us^ of an info^tiotf gathering method is more important 
than the amount of information expected" (26,27). ' 

The rapid giyjwth of technology superimposes upon this negative 
attitude the fS^ct that constant' training is required to use the newest 
tools to advantages The information Intermediary has an increasingly 
Important function in liaison between the service and its potential' user- 
cusjomers. This is one element in the" question; will it be used? Does the 
customer want to use it, does he know how to use it, does he need help to | " 
use it? (26) "Information broker" is the new word for 1976, and additional 
new titles can be expected this function becomes more 'complex. Repeated 
efforts to train the customer himself to use the more sophisticated services 
have su^ested that the person most likely to look for help is the e»perienced 
customer, who knows the value of information. No matter how much or how 
little such training the' scientist may have, the information expert has more. 

The inherent conflict beti?een quantity and qualify of information 
is made worse by this division of effort. The tedhnicAL function Of pro- . 
ducing references is the work of a specialist, and the evaluation of infor- 
mation ^s the job of somebody alse. New technology produces a large increase 
in the quantity of information to be processed. Without an intermediary, it 
is the customer s tiifle which must be used to reject the trash before the 
"information" prqduced can be used^ The quantitative selection is by per- 
tinence or relevance, not by "value." The th read is tfiat had data will 
push out the good. This tends to repress the^Bfepted value of any single 
fact, and if ten bits ^f information are not enough, it is possible to 
produce a thousand. To meet this threat, it is vital to recognize that 
both the operating function which produces the information and the 
research pr planning functions which' use it are necessary and independent.^ 
Neither can^ function properly alone. " ^ \ 



Er|c • 64 



- 42 - 
APPENDIX B 



DEVELOPMENT 



Proposal Postulates 



Throughput this project, the Identlf IcatTon of varia1}led con- 
sidered as significant by. different users In thelt selection of Informa- 
tion services has been based on one*-to-one interviews, in the field* The 
survey and >lt9^developlng conclusions were discussed with each person 
on the average two or three times during tlie year, for periods of about 
5 to 30 minutes* T^e object was to recognize and discuss the working 
bias of each user interviewed, give full acceptance to Its validity, and ^ 
thereby establish it a9 data for a generalized apc^oach. > ^ 

* Several postulates to be examined were I^k^ forth ifi the proposal 
for 'Contract C-1027.* Others .were developed durin^pRie project*' The 
following initial postulates were drawn, based on several decades of 

experience in information research: . 

I' 

- There are identifiable groups of users of Information 
services who differ from each other in their basis for" 
selections between competing services; ' ^ 

" The examination of different, points of yiew can be far 
more productive ifi the analysis of a^ information 
problem than .an e^diaustive study of any one viewpoint; 

- The many factors used for cbmparisbhs tg&tween services ^ 
^ are interrelated, «nd princlplea can be foun#%y which 

they Iqlpract f<or different categories of users ^and * 
different types of services. 

^' Statemenrts on the relative values of information 
services In the literature or elsewhere must be 
considered In the light of who says , what, aiid when; 

- The consideration ol who says what Is in on^^ sen^e 'a 
direct opposite of the '^Delphi appro^^" which madks' 
personal differences in anonymity, on the^ assumption 
that a statistical consensus 1$ more useful* 



Initial Interviews . ^ . , 

' — • - ^ ' * • . 

^ The first 'interviews were conducted with l)Etdlviduals who 8pe^l^ 

from 50% to lOOZ of their time in the* use of information services, to look 
for areas of agreement and disagreement in how^they select which service to 
use* These interviews addressed two ^estionss (1) What specific 
abstracting/indexing services do you find most useful, in Vour daily work?" 
and (2) "What discriminating, factors would you recommend ^''\ti^lllng someone 
how to choose between them?" ^ 
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Four full-time searchers were Interviewed, spearately and together; 
their conclusions were in good agreement, as siimmarized in Table B-1. 
Counting related publications by a given publisher as a single source, 
they^ named a half-dozen services as "unique" for their use, not replaceable 
by anything else. The second column In this table shows the discriminating 
•fdtetors they selected as the basis f or -compariwons. These were used as ^ 
column headings to expand the table into a grid, with entfles for each 
service listed. ' v ^ 

^ No attempt was made to e^and the obyious heading of "specific 
fields co^red;'^ Upon further consideratipn this variable, important as 
it 13, was excluded -from the study by definition: if tjiere is only one 
service covering a given field it 'gets all the business the»e-is, because ' ^ 
.there is no competition- If there is competition, the, choice is based- on 
other factors (completeness., J^uklity, etc.) and not just, on the field of 
coverage. . , 

The column for cost was left largely blank in filling out this-' 
grid, even by the most experienced searchers, they referred to the staff 
administrator in charge of budget for more accurate- informatiog^ The same 
thing happened repeatedly in other interviews- whenever costs .were mentioned. 
While cost is a fS^or in Initial-purchase decisions, the choices that 
turn out well seem to involve an essential component of experience , or,,- 
the manager *s hunch. Once the service has been purchased and on the shelves^ 
its purchase cost :^s no longer ^^considered as regards day-to-day use. 
Operating costs* become a large^; variable ^ and it soon became apparent that 
cost is a factor that 'is comij^i^y deferred to someone in management. A 
number of the discrlminat^^Sflkctors named are jpel^ted to^phys-lcal facilities, 
^pace, and the time Teqxitii0 M keep the service usable. These are aj.1 
budget items, related to inMfect costs. 

In developing Table B-1 it becMie apjJarent thajt the variables 
listed as column headings deal with quantifiable factors, while quality 
factors tend to .appeaoc^.in the comments. The deferra! of part "of the 
decision to management.was interpreted at first as based on costs and . 
budgets, but (Conferences with various managers emphasiaad a strong * 
experiential component in management decisions. A typical statement was: 
"If I were consi<fc^|lng a service where I had no experience, I would try to 
find another mani|j^fe>^whom I coufd trust* who did have some experience I 
could consult." lift^p^ioral analyst consultants helped identify, this 
dimension as'^he judgmental/ Intuitive aspect in the decis:^on process. -Thi? 
aspect is charact;eristlc of management, but is* by no means ^o 'limited. 

T^^ases^ for comparisonJl^Mn- Table B-1 were n/extr expanded to a 
^^^^jH^ :0^p^^^9 grouped in the 3 dimensions named. One draft of this ' 
is shown ;iih^»lfe B-2. The lines between categories are not always sharp, 
•since there are some factors which can go In more fhan- one place, but the 
pattern can be clearly recognized. Thft Tabrle B-2 was n^xt shown' to seve^rai 
sciantists who are frequent, customers for infon^ati'on searches and other 
services* They agreed on the significance^ of the factors ajid dimensions 
named, but disagreed on tl;keir relative Importance. The s'bientists all 
said that they would put qualitative .factors first in their selection; 
"Whltih service is a. pleasure to use?", and pay. relatively little attention • 
to* the ^qUantjLty factors as a group. - 



TAgLE B-1 

DISCRIMINATING FACTORS APPLIED (4 USEfeS) 



"Unique*' Ser^ic^s, 
Most Used 

Chemical Abstracts 

Inde'3^ Hard Copy « 
Text Microfilm 

Derwent - C.P.I. 

API - Abstracts: 

^ Patents and 
Literature 

Science Citation Index 



Predlcasts, 

CMA, Funk-Scott 
Engineering Index 

9 

Applied Science and 
Technology Index 



Bases for Comparison * 

Fleld^ covered 

Cost $/yi^ 
Shelf space/yi^ 

Timeliness: text + index 

User requirements 

training 
skill ^ 

Minimum use required 

Maintenance time . 

Comments 
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• ■ V . ^ , ' TABLE^-2 ■ 

■ , V _ • ' ■ ' , 

• FACTORS.IN SELECTION BETWEEN COMfE'^NG^sfeviCgS 



Dimensions In the ^^lectloa Pro6e8s: 



(a)/, qupntdtative factorf 

« nimerlcal 
(objective) 



(b) qualitative factor! : . 

^ rankings . 

' 0, (includes subjective) 




(c) husinesQ judgment: 

» go/tio-go 

(Includes Intuitive) 



/ 



* cost factolfe, (for f;lrst look oy continued 

CQ|||||ge of selected fields • ^ . 
c^leteness {no. of documents in field> 
(Z of available literature In field). 
» time factors for delivery « 

(soonest and latest, as well as avefage)' 

for&t of report * 

- appearance 

^ ^ orderly ' presentation • V % 
V - convenient ^ccess within report, 

indexes, arrangement w . * ' 

flexibility of* coverage i ' . ' 

- abilitx to vary- (+ select) ^ 

depth of cove^ge, details - ^ 

- ability to highlight sfelected items « 

by-closenefi|p of mat^h to question 

. ^ ^ ^ : 

responslvenessf ^ ' ' ' * # 

- can it change priorities 
add -new items oif^ request 

- tolerate a*tguities y ^ . 

- will suit, '"our way of doing tftlngfi*': 

feedback * / 

commi^icatlon with, ^r by user 

- to redefine quegtlon, during search 

- siiitability for browning 

reputation - * 

-^recommendation ef vpast useifs ' 

- reliablllt]^ uniformity of}quality 

- oonf identiaiity, . \' ' \^ 

- viabilitf, will, it last? ' • ^ 
• familiarity, to; improve efficiency 

factuality * - ^ * ^ 

- if he says a thing is so, does be * ' 

have a basis for it? 

cow's is tency * \ 

- recognised dependable bias ^ 

experience in actual type of work inyolvijd 
- (nb^Just .the field) . ; ' 
* skill to anticipate problems 
T optlmizatioii of prodedulres * , * 
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: / ^ ^lii;:'^ Vi^' ' Vlhe three-dlmensioMV^utline ^nd thi^headlng? In TaUle^B-l were 
neict checKed independently witik -the. managers of ,two outside information 
£ftije;er8.-*a pharmaceutical^^g^rch library and university sclenc^ 
^ *C V library. They conf in^^ tW^Tieadings selected aftd the same list of* 
^ A services- ^They„ reconiiended the BIOSIS and Medlar s/Medline ser^ea/rfs 
"unique^' .dn their field (replacing the API abstracts) wi^h thejjKatlfLon 
that initial training time and regular continued practise are esfential 
to mait^taln'the ;niniiaum skills required for their effective use. Overall, 
ifc* * the lists of services and dfscriminating factors obtained independently 

** • from these interviews showed /a high degree of agreement, with minor 

'differences in the ordfer forUpecifie individuals, 

\- - - \ , - • • 

In*an attempt to generalize why -this agreement was realized, it 
appear^s yiat these are Services where the same amount of basic training,^. 
Bxperience^ and facilities makes it possible to answer more different types 
questions than a competing service, which, is rated valuable, A 

ferent concli>^ion based on the same data could be that because these 
ice^^-'arei^used more pf ten, the users became more skilled in them and 
them e»ier to use. This is not necessarily' a different conclusipn, 
er, since both may be related to the observatikjn that this- service is j 
flexible, for tore 'lisers. * 

No* such agreement appeared ir^effprta to prepare a 1d roarer list * 
of this type for the same users, to ranK the 30 to 50 abstracting/indexing 
services they consider most\juseful: . tL^les and rankings beyond tlie few in 
Table B-1 were cqpipl^ti^ly different for^each user./searcher , or for the same 
searcher for di^erent searches. This suggested a further analysis of the 
data to i^Dnsider what types of* questions get the widest fluctuations from 
one worker or one use to another, apd what questions^are fhost* likely to give 
a bett'e^>^^c^ing of answers. The "unique" J-ist was probably a good qiiestion 
. *. on .this 'basis, and^the second list was not. The failure of this Effort led to* 
a newr.direction in the study, to look for areas or circumstances of pre- * 
^* -die table agreedlhit or disagreement: in the basis of selections. Stated 

dif fereptl]P, the important varia*ble is not just the user as an individual, 
but a combination of usei^ ^||Laracteri§tic8 with the specific use at h ^ 

* Perceived values in the importance of.tfirect and 'indirect l^sts ^ 

are^n obvi^s area of disagreen^ent* Subsequent interviews ij^dicate^l'that * 
, time is anotljer, variable with aspects which differ greatly in tfieir relative ' 
importancta to different users, Th^re is a qu'^stion of whose time is most 
valimble ^nd wh€n, wha^ portions of the total information .t^equest the customer 
is willing t6 delegate at ^a given time to som^pne else, and hoW'much of 
this work the customer prefers to.' do for himself, ^o become 'more famiMar 
' with the material. These were examined* fur the r as interaction effects 
which involve both the user and the environment of use, ^ 

User Population Groups , ; ' , a*^* " ^ . 

This .confirmed the original strategy decision to examioe as, ^ 
carefully as possible the nature and reasons tf or individual bias and 
report them anonymbusly at the end rather than concealing them 'throughbut, 
so as^to develop^a fuller understanding of the decision-making process. 



ERIC — . ' ^ 




\ 

The revlfew of areas of agreement and disagreement foiraS^li^. t^e 
initial ihterviews suggested a more detailed isurvey of usage patterns aiaong 
the 15-20 regular ^users \Dr information services at ^the Exxon RBsearch and 
w^ngineeting Central Library who spend some 50Z or more of their average 
\orking day in aearching the literature. A third of* these' are library 1 
staff » half of fl^' information analy^s/or searchers who have special 
areas of interestt and others Include scientists who are strongly priented 
toward the use of the literature. All of these groups comprise ^^Vlduals 
with different levels of training, skill, and responsibility - i£cXudi^ 
the freedom to decide, which searched are most worthwhile, anid 'whart informa- 
tion will actually ftte usefl. 

The first step in the survey proposal was to conduct^ preliminary 
Interview with major users, based on their records of past experience, to 
idet.ermlne what information servicae ea^h one uses most and abo\it how often. 
This was to decide what question^^ are worth asking, and what serviciss 
should be added or removed from \he previous lists for tabulation. Pre- 
liminary indications were that mi^jr users hav« on^or^two favorites, which 
this individual uses more than alX^ther services put together, ^ (e.g., CA, 
Derwenti abstracts).* The plan was to ask for a complete tally*'f rom " 
,^each user for s^ver^ weeks, with follow^ip discussion, after eliminating 

* for each individual as a convenience those^services used most often 
(i.e., 30 to '90% of all uses). Xhat is, the survey was to be simplified 
for each participant by identifying this pattern ,in the preliminary inter- 

fview, confirming it, and concentrating thereafter on incremental uses of 
other services. 

. This ,set of prellminarrrt^terviews led directly ie the identifica- 
tion of the'^'user populations" referred to above., At'^^the tiipe- the survey 
was started, the distinction had been' drawn between Searchers, scientists, 
arid infonhation managers as user groups having different sets of t>i^iorities. 
Completion of die initial interviews identified the se^f€n ^ub-groups of 
user populations discussed in the report. Table 2, and their differences ' 
in value priorities as reflected in Table 3, This filled jgit two elements 
in the Mat/ix proposed, and redire<ited attention to a cfOser study ^f the 
characteristics of^tLe^user grotmpWd possible interaction effects between 
the- type of user and the type ojuse. A more immediate' problem was the 
question of acceptable defdfliitions for user groups, and wlfere to put the 
"manager" as a user rof services. . . " 

Management Viewpoints , . \ 

Ingm^ovements in\the use of scientific and technical information 
constantly run into t||e limitation that the number of managers who tare 
greatly interested in information systems and services remains about the 

* same, regardless of the changes and improvem'ent's made. The 'original scope 
of work^of this project and initial conference^ withNSF emphasized the 
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taportaace of getting inpuf fVom different maflagers*amd disseminating the^ 
results widely to the attention of raatiagers oujtside of the infptmation^ 
field Itself. T^is proved, to be a prof itabl^';approach. * / 

The original ^outline\of. djlmenslons was dispussed informally at 
the Engineering Foundation Conference o^ STI Services at Hennlker, N.H. in 
August 1^75^ and expande<^ at the AglS meeting at\Boston in October 1975. 
Two Important elements' lA the selection process were reported in numerous 
user interviews as^ beiflg "deferred to ^nager"4 questions of relative co^t 
^^irect or indirect), and the preparation in advance of* an approved ,14,st 
ot services (based on costs or other factors). Th'^ operating "matlaget" ' 
was considered as a- special user^group, with additional fu^ciions^ such as 
project evaluations, and the choice of a wort environment. - 



draft of Table 3 on* the priorit4||^p^€€^Ved -for ' selection 

j" w^ 



♦ A first aratt or Table J on 

factors in different dimensions place^ "managers" w^Pi "vendors" 'as ^ us^r 
population to whom costs* come first, whereas "^scientlst^' and '"planning 
staff" put their emphasis on factors of Quality and exiterieiiee. - TJils scheme \^ 
was not satisfactory, since the manager's viewpoint can be just a& clearly 
i'dentif led with, planning sta^f ^ a major fur^ction.. It was changed acc6rd- 
ingly in an Interim, Report (MacA 15, 1976) prepared as a popular pre^enta^. 
tion of the "Matrix of-.user^vdraes and interaction^" to meet editorial 8ug-> 
gestions by the editor, l)f .the ACS journal CHEMTECH. This article (29) was 
publish^ in-August 1976, addressed to '^-broad audience of project m4nagers 
in chen^lcal technology It emphasized the impact!^ of growth in information 
technology (see Appendix A) and analogies b^twee^ the sele<iti0n of 
engineering services and in^rmation service^b. A ' * ' ,^ 

The Matrix was. expanded tcf include user impacts as an additional 
interaction effect,^ and y/tts presented at .the National Information Retrieyal 
Conference in Philadelphia, in May 1976. Discussio^i with leading experts 
in ^he information literature at that ^ time confirmedche dimegpional analysis^ 
and matrix ^approach 'presented as valid and useful. 



matrix ' 



The presentation for NIRC wa^ first previewed and approved by an 
Oversight Review^ Comnittee meeting at N^F the end o^Apr^L 1976. J^lans were 
^also discussed at this review meeting for, a questionnaire to expand the 
data base of replies from initial interviews. This had been designed for 
use in-hand, during -follow-up interviews, to correct for the initial bias 
of replies obtained most easily from the "movers" ^who '^re most interested 
in new ideas <see ^^ppenftix Q) : During this discussion,' particular ^emphasis 
was placed on the observation that,, .the diffjerences d.n, value systems of * 
different users^wOuld also appear differently to th^jee different groups of 
managers. NSF] directives^oll9wing this* discussion urged: (1) ittore att^tion 
to the differences between managers; this has 9een expanded m a major con-, ^ 
elusion in the Executiv^^^uimnary and JeJtt Table &,jand ^X*to test the^ , * 
inroposed questionnaire^ a larger sample^ to exam|^ the' "value px:etfer^nces" 
of different user* populations. « This test ha^ been made,;^wlth proc<»djures 
and results a% sttmmarized in Appen(lix*C« ' , ' ^ ^ 
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The vlewpoiiits of different managers were explored further during 
1976 In rfnterviiews-wlth selected indiV|.duals at the ASK mid-year meeting 
at Nashville in May, and at the Gordon Research Conference on Information 
Problems in Scient^ific Research Ici July, as well as at NIRC tod the Over- 
sight Review Meeting in AJ)iMl. The general trend of these comments* has ; 
Ijeen wqjpked into' the discussion above (Text Table 6). The follo\Ang are 
highlights , of ' the comments of different managdra consulted: 

^ ' ^ * : ' * 

- Prefer^nca ranki^gs ,in the Value systems discussed are 
relative, nof absolute: ^a low ranking does not imply 
zero value, but only that other valjiies considered * 
more Important. ' > , * - 

The fact that. an. item of information is liew to the user 
may be enough for some'(R/D planning, marketing) ^jut not 
at ail'fc^ others <^atents,. contract, negotiations) ♦ * 



- User-impact interactions may have an effect on revisions* • 
in t:he management de^sion as to how much time can^be 
devote to looking at Infonaation rather than other work 
(such as the effect on usef sel^-esteem of paying someone 
else to do thlnk^Jig for- you-, father than simply looking 

.up facts), \ ^ ' ; % 

vr, The^balance between direct and indirect co^t ef fect>s\as a 
' major Interacticfti with time consti^alnts, which differs 

- with the. type of ijset /manager and the environment of use. 

r- - When servl<Je'^^.eatJ#f actpr;^ x« foit'the service 

%^f^3. whol^ may- beo much ijiore eaSfly^ recdgai^gd than "critical 
ihf i'dent" values-, yhich seem, to cause conflict in claims . 
for credit \ ' , ' - 

- A dfesiral^e end result of#the present approach is ^ sist oi 
classificat^ns by yhich peonle' can find a home "for their ' 
viewpoint,, Vlth-segm|pts froir^e user end ra.ther''than, 

, as 'usual, fr*m the. mapager or prljjivider of the setyicef 

" • ' ' ' ' ' ' . • ' . ' 

- Individuality is difficult to. promote^ In any bureaucratic ► 
system, and 'functions which, require ir. are likely to be 
dowrigMjded or unrecognized. 

.'-'-''^^ ' ' , ^ . , ' 

- 1 Per sod^ skills pf indivldu^U-S In the system are clearly^ 

recOlgfC^ed whenever special services for special cu8tt>mer 
are de^red.' ^ ^ L 'J t , ' 

- Jiidgkental ★a^ues are not well appreciated by the **tationaI 
^ man*' who expects his system to be all things to all 'men, If 

only it can get^enough support.*/ 




- Planning Staff nevfer^has ^xtra ti»e, has'access to patterns 
of 4pcisio% ntft available to the scientist , jand is more 
concerned vith plausibility of a point of view, and meana 
to evaluate it, than where It comes £rom; time means money. 

• * « 

;^ " •The university environment is^very different from the 
Industrial one,. with cfheaper labor 'and fewer time con- 
straints, but strictly llarlted capital. Industrial systems'' 

S* ' are far more sensitive to time'/labor costs, which depend 
strongly on whos^ time' is being spent. • 

m ■ ' 

- A major management concern before purchase of a sa^ice is 
. to be certain that different bases of decision have' at 

^least been recognized, and not .overlooked because o^ a lack . 

of extensive data. » > 

» » - •' 

Managers and User /Workers^ Agree in Function ' * * 

~' ~ * ' 

On furtfier study of the' priority values of managers in field 
interviews, . it gradually appeared^that the word "manager" could ^be placed 
Just as well one place as another in Table 3. Every user popxxlation 
includes manager/uders as well, as user/workers, and e^^ry user»can be a 
bit of a manager. As- the initial confusion was resolved, several con- 
clusions appealed: ,44, 



Different groups of mafna'gers can be found, just as with 
othe?!^ groups 6f users, who differ in the values they 
perceivq for different factbrs in their selection of 
services. ' i'' . • ^ 

The difference lies in what the information service Is 
being used ^for at the moment: the research function, the 
planning function,* or information operations. 

Managers and user/workers ^nvolved in the sane function share 
the same relative values for the prlp^ties perceived, and 
the/ same set .oT meanings or connotations for Che yords used 
fcMT key parameters "in the selection process. 

AccMr(jn.ngly, Table 3 was changed to refer to user functlpn 



generically, w^pt than to specific user' populations. Text Table 6 ox^the 
conflict in ! cmS^ation^ for key wor^s was dejf^loped to c^iatdcter^e the 
differences d|parerit to different managers,. ^Here too^the same' differences 
. apply .to- both s^nager and, w6rker for a given Informat^a service function, -n, 
. tt is the function which" contjrols, so the word "manager" was dropped from the 
caption of thfLs table without changing its content. It may noted that 
several of th^seTVD^ds can st^^ll be acceptc^d as terms In Text Tattle 1. 
(tlmelines's, time, depth), because tt\elr connotations in conflict $lU. fall < 
intone dimension. This, is not true Jor the term '^reliability^ whic*was 
rejected as ambiguoui,* because it ma^ refer to diffeirent factors in time . 
(quantity.), in- uniformity (quality) t or in reputatrtcwi (judg?»ental) 

. , .■ ' .■ ■ ■ -■ 
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The unifying character of this agreepent In function Is significant. 
The fact that, the manager/us^s and user/wprkers see things the same way f rom ^ 
this vlewpolHt appeats to vAlldA'te the concept of user functions, just as • 
the disagreements a^d agreements on prlorltlA of different' user-populations 
appear to validate/the concept of dimensions In selection factor's. The 
firtcllled manager any user ^applying management skills can applV the 
dimensional approai^ to . Identify are^ In which he is strong and not 90 
strong, and delegatfe or seek assistance^ in the weaker areas to improve 
cost ef feotlveriess tfor tt^e same-tota^. effort. v 
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t APP^M^ C 



^VEY QUESTIONNAIRE 



Variables for Analysis 



^ The analytical piiocedures and intervHw approach discussed in 
Appendix A and Appendix lied to an initial Matrix containing three 
variables aftd three interadtion effects* These formed the basis for the 
survey questionnaire. The , first variables outlined were the ciassiflca- V 
tlon of selection factors into dimensions (see text Table^ l),-user popula-' 
tlons into groups- (text Table 2) and information services into types, as 
they Interact with the user (see Definitions)* The differences perceived 
Were sharpened by the concept of priorities in user value systems as an 
interaction effect, characteristic of the user group (see text Table 3). 

^ Ar second ilxteractlon e^ect characteristic of the type of service 

Was Ident^ied as the personal impact of the user paying someone tq do 
thinking for him, in inficrmation aiuilysis as contrasted to more routrtfie 
sewicjBSi This was ^generalized to a desire to know and choose a speific 
individual, whenever the user is selecting an information service whMh^ 
he perceives as involving his own highest skills. This could be searching 
for the searcher. Indexing for the indexer, or analysis for the analyst. ' . 

Differences and similarities were accentuated in the interview^, 
"pertiitbatlon analysis" (see Appendix^A), asking e^ch person who agreed 
with a glvtn corf elati5||^ to identify others who would either agree or. 
disagree with his point of view'. The next interviews sought to work out 
with both parties a suitable working vocabulary to identify the differences 
they percelvetd* ^simple *set of definitions acc^table to people who 
dlsagree'^on priorities and values is far from easy- to achieve. 

The next^step *was to examine each area where an j^e'ractlop 
appears to exf^t, and try to find wor^s t0 describe it. This identified 
a /third int (Br action effect^ user poptil&tlon'&^whoiagree in their ranking 
oi priorities in selection facfbr value also tend to agree in the Importance, 
, which they attach t;o hAvlng the originals of Tselected references in hand 
for their own use, rather than abstracts^ titles only, or a iflgest review. 

Further donslderatlon of the procedures used to reach this stage 
fli^gested a danger that; the results pbtaiijed could be influwced by subtle 
changes.jin phrasing the questions addressed dtT different u4%r groups. Each 
intetvlew for the analysis of ,var labia's covered ^e same general area,i but £ 
the emphasis kept shlfttng as n^ ^ispri-ela tlons were found. ^This ^led*to, ^. 
plans for a standard questionnaire to be used in hand for second-ground 
interviews, to get a clearer' view of i>the differences between groups when* 
the same questions %mre asked in th| same way. 
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Prellnlnary Design of Questionnaire • f 

• ' ' ' 

The first plan for a formal questionnair/fe vaa for a usage suryey 
based on the concept of different priorities for different users (see text 
Table 3) . This was to be addressed In-house to some 20 heavy users of 
information service, from different backgrounds (see Appendix B) . The' 
^object was to examine the concept of priorities'.and Icfek for qther variables 
in the selection process. The table of priorities at this time listecl tfi^*^ 
names ol- separate user groups, by specific discipline: scientist, engineer, 
patent attorney, searcher, information analyst, manager (as the on§ to whom 
cost decisions are deferred), vendor (as an author interested in dat# on 
cost effects). Comments on the , concepts of selection-factor dimensions anS - 
user-value priorities had also been obtained from separate surveys of the 
users of a small special library, managers of InformAion services at an 
outside conference, and* the sources selected foj? 300 patent searches in 
special subject areas vdered by a group of patent attorneys over a pericni . 
of 2 years. ' * * ' 

Requirements proposed for the usage survey quest Amaire were 
first to identify the user, in terms of dis^plihe or user population at 
the time of a specific use eyent. Thia wiuld be followed by a simple grid 

^or each user with entries for each special eVent, and variables arranged 
as column headings to be checked for a subjective ranking of perceived 

. Importance. The format was to "include, six variables for each use or 
user: the three dimensions of selection factors, and 'the relative importance 
attached to recovering original documents, abstracts, or reference |)y title 
only. Additional variables might added. The idea was to simplify 
record-keeping for each user by^ a preliminary interview covering these 
questions completely for those services Which he used most of the time, ' 
and ask him to check the questionnaire only when for some special reason 
he used a different service. ^ , ' 

This format was modified and not used after the preliminary inter- 
views, because it was clear tiket it placed the emphasis in the wrong place 
(see Appendix B). There were repeated indications that the ranking of 
faptors for the selectipn process depends upon the type of qiiestipn or how 
the •information is \t be used, and not on the user's discipline. One such 
indication isjh^ow easy It. is to design a loaded question which will make 
any desir^one of a set of services look best, then ask other qilestions 
designed to complej^ly reverse the order. This game can be played with 
any of the leading Zipmpeti tors, and it id a favorite pioy of the salesman 
for a new service. * 

« 

The original postulate *that usj^ge patterns could be tied primarily 
to the IndividMl was thus di«catded. Further attempts to set up correlations 
with service li^st used," subjective rankings, or ^ield of ii\terest were 
barffed more on studying the use at a given n^nt. Unfortunately, this com- 
plicates the problem of ^ the definition of/ferms which do not mean exactly 
the sa^e thing ±n one use as they do f or|janother. This was iden£lfied m ' 
a cause of ambiguity, in the statement off priorities for the "manager" and 
!tesolved when it \)as recognized that here, too, the *cqptroiy.ng variable 
is not the person or his job title but his use for. the service selected". 



ERIC , , 76 : 



9 



The questionnaire strategy was redesigned on this basis to develop 
a standard set of definitions for variables and priorities from the view- 
poiilt of use rather than the user^s discipline. The same set of propositions 
were to be-presented in tiand t^ the oijiginal cooperating SAt of 36 users 
for re-interviews, including any modifications made after/th^ir first 
comments. Personal Impacts between the user and the servxce were added as 
aij interaction effect for study, based ^on observations that the users of 
information analysis; patent attorneys, or planning staff want to selects 
a searcher they know, even IS- he moves to a new firm. ^ " 

An outline of the foijnat proposed was presented and approved at 
th^ Oversight Rev^lR^ "Committee mieeting in April 1976, at the ninth month of 
the l-year contract period* The Project Director and the Committee sug- 
gested that raw data of greater Interest might be obtained in a limited time 
by sending the questionnaire to a,<larger sample of the populations avail- 
able, instead of using it in-*hand for re-interviews. This sample should 
also help to reduce the lilas due to getting initial replies only f^om 
individuals known to be actively interested. \^ 

Filial Draft 

A draft of the questionnaire was discussed Internally and reviewed ^ 
with outside experts at NIRC in May, with particulate attention to questions 
in the dimension of budget/costs. . A second draft was prepared and presented 
informally to individuals and interested groups at the Midyear ^SIS meeting 
at Nashville. A. number of suggestions on principles and methods of cor- 
relation were received from operating managers who had conducted user sur- 
veys at Battelle^, Rand, and Rutgers. A helpful line by line review was 
also ^conducted in a breakfast discussion with a group of students from 
Syracuse University^ 

' These suggestions were combined into another draft and reviewed 
with , a consulting behavioral- analyst, who recommended a final check with 5 
"guinea pig" critics to take the test without oral Instructions. The 
results from thia^ suggested the clarification of deflhitlons and instrucf ions 
at several points* . 

The final form of the questiohnaire is attached hereto* This was 
deemed ready enough in this form for' internal use, where the returns could 
be followed up easilyv by- telephone. It was recognized that farther changes 
in format, would be dedired If a similar questionnaire were to be used 
again. The fortf Q is in four pages,' with contents a? follows:; 

\ ^ ji • . \ . ' 

Page 1 ^ preamble with definiticm of tenns, expanded. after 

thk "guinea pig" test to minimize misunderstandings. 

"anonymous" definition of user background 

^ frequency of us^ for three types of search request 



Page 2 - frequency of use, for specific named services 

- ranking of selection factors and costs for named 

services known to the user 

Page 3 - relative Importance of selectiod factors by types of use ^ 

- Z of references wanted as originals, abstracts, title only 

- opinion check list on personal impacts and preferences 

Page 4 - open-ended questions, to help counteract the problem of 
forced definitions and oblgultles 

Questionnaire returns show 100^ replies to tne Identification 
question 1, 78Z provided the optional signature, and only two persons * 
removed the mailing label ior anonymous reply. The r$OBt ^effective 
questions for statistical review were the frequency check lists ±n 2, 3, 
'5b, and the specific preference list In question 6, The 'two questions 4 and 
5a on the relative i;anklngs of selection factors were too complex and read 
differently by different respondents. The concepts of text Table 6 were 
developed from this experience, to clarify importance of the functions 
identified and their different characteristic connotations for the same ^ 
key words. ' 

The open-ended question^ on page 4 received an 85Z response, *andv 
over h^lf of these replied provided some detailed comment. 

Procedure and Initial Results 

* 

^ The sample chosen as the ba^s for test was a random list of 
240^ named individuals picked from the professional employees at Exxon 
Research sind Engineering Company. In this sample 130 individuals were 
located at the engineering research center, 90 in laboratory research and 
20 in smaller groups such as information specialists or patents. These 
sub-sets represent 10-20Z of the employees sampled. The list was then 
checked or a telephone call was made to ask: have you during the past 
full year (1975-1976) done any searching in the published literature apart 
from reading current journals, either yourself or by asking someone else' 
to conduct a search f or j^ou? The questionnaire was sent to a total of 89 " 
who were known ^ us^ti or who^fhus identified ^hemselves. Response to* / 
th^ form took 4ii average of 15 minutes. 

. ^ Ihitlal query showe4 thatv49 of the 130 engineer^ used avrfll- 

Wble l^tejrature services to search for information during the^ear, versus. 
Mr who ^ea^hed only in company Sources or current' journals. The same * ' 
ratio, for fe^^ch was 40 to 50, or nearer* 1 to 1. The list of 89 ''known 
users" amounts \o 40Z of the total random sample of 220 from thesfe two 
locations. This in^na that the other 60Z of the. test sample do not consider 
these information sexylces as important to them* They satisfy their infor- 
mation needs'ln other ways, i.e., from original journals, company sources,,, 
or their colleagues in the invisible colleget as Ipdiclated elsewhere (13,30). 
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Table C-1 ' 

* « 

Questionnaire Returns (Raw Data) ^ 

''ff ^ ' 

. Engineers Research 




Total Sample 

used literature services, lS15llf> 
not used during 1975/76 

Questionnaires Returned 

Number of Searches Requested 

0-1 F 

2-3 

4+ ■ >; ■ 

Number Services Specified 

0-1 
2-3 

Ranking of Selection Factors (replies) 

quality/reputation/costs 

all equal . « 

costs first ^ 

Pre|pr Originals to Abstracts (or reverse) 

for R/D . ; 
f pt reports 
/fbr decision • \ 

Pefceivetf values (ai^r^ge) 

khow the individual^" , 

use' known service • 

know appro xifliat^costs \ 

compare on exact costs 

computer-customer training 

accuracy > completeness * 

prompt key refere»ces ^ 

Open-ended co^iment s 

total ' e - 

detailed ' 



130 


90 


AO 


/.A 


(51 




33/49 


38/40 


J* 


• 


19 • ■ 


■13 


5 


8 


9 

• 


V^15 




• 

■ 10 


9 


.. 15 


7 ■ 


^ • 13 




- (30) 


23' ' 


• 2&-; 


4 


3 


2 . * 




. V 


• 


'+3; -15 


+12,-9 


-A 


.+1,-8 


+1,-5 

• * ^ 


-5 


c .-.24 . 


+. 2f3 - 


\-.06 * 


+.30 


+.25 ■• 


• +.06 


-.72 ' 


*.67 


-.76 • 


-.77 - 


.53 


.41 


.79 ' 


•91 


\ ' 30 


50 



17 



16 



79 




/ 



The "greater interest of .the researchers In Information services 
,1s' reflected In responses received, which was 38 out of AO from research' 
versus 32 out of 49 -from engineering. 'Similar • trends appear lit the' ' "'^ 
average number of searches made during the yey", • WhkdJlJ was about 2* for 
• the -engineers and over* 4 for research.' The same applies. to/faad.llarlty 
wltli different information .services; a, check Hat by title" pf those used 
at least once during this period showed an average pf xxne for the en^lneecs;. 

and^ three or more jEor' research. * ^ ' # '* , v ^ 

• . . ' * * . 

A preliminary table of questionnaire* returns is given in Table- 
C-1. -These are presented as raw data, since Contract C-1027Jrequ^ed only 
that a questlonnalte be pre par^ suitable for* testing* and avowed no time 
or funds for st«t 1st IcaL analysis^ The arbltrary'^dlstlnctlod between > 
"englnfeers''' at'th^ englneerlhg center location and "research", at the 
research center shows a number .of interesting correlations. Both groups 
agree that factors of quality .(convenience) and judgment (reputation) 
^re ranked *as most impotjt ant (4 ot 3) In th6l"r selection of * services; ' ' * 
where time anfl^ cost ^ factors were ranlced as\almost nil (1 of 0) . " . ■ 

, 'Different values perceived by fetfese two groups were ranked pn." 
a 5-pofnt'^cale (vital,, yes ^ neutrar,«not needed^ and prefer not)>. .The 
algebraic average v^fi strongly positive -for research and negative for 
engineers on the use originals versus ai)stra<vts, using known ^sejrvlces, \. 
and a^lng a known individual to collect the Ittformatlon. . Both groups 
fgr^ed (90 vs. ^80%) that it is Important to g^t key. .references promptly, ' 
bef ore /any fu^Jher searcli. "^Bpth ranked compl^eteness in a»searth. ahead) ' 
of accuracy, the engineers a iittle^mofe so'. The^ strongest agreement/^ 
was on two questions with Kfeavily negative r^spon3e and' 10-20Z the - 
total as "prefer n©t,^'-for the importance -of knowing e^act dolla^ costs 
tQ compare 'Services, and even more against expecting the user/customer , 
himself to. know hpw to do compiitet sear<:hlng;. The two -grouiJs responded' , 
equally well, to the request for comments. * . • 

^ "The preference for using abstracts ratrt|r thaa original^ for 
purposes other than research was^ strbng^r thkn» expected and it seemed to 
Increase for uses further from research, for report writing or for liecisions. 
This preference inaj^ reflect a Ideal bias-^at this location, whei^ an aggres- 
sive effort is made to provide, good abstracts di:fecte'd Ji|gpaterlal of 
coiApany interest. I 

. fhe differences observed between the userrhabits of research ' 
scientists and eil^ineers checks a survejy of the same group pf engineers 
made during a previous year, when 100% of the library requests received / 
for literature references came from 41!^ of the engineers. Both research 
engineers atid design engineers were included. Requests ranged from 25% 
of "light. users" averaging 4 requests per year, to less than 1% of "heavy 
users" (5 men) averiaging 5 tequests per week. These are the "information ' 
gatekeepers," only 1 out of the 5. on senior, staff. * This .Indifference to ^ 
the literature 4iay be a -factor In- the Zlpf-law resultjs reported by Allen 
among elect ifibal engineers at MIT. it should be noted again in this con- 
pectlqn that in contrast to scflfnti^ic 'jdornais and reports, 'the engineer 
does not typically expecp to find answers to his problems in the publj.shed" 
literature, and it ±t unusual to %nd^fen|lneerlng know-how in Written form. 
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Return To: 



Homer J. Hall ^• 
Govt. Resea^i Labs 
ERC-28, RA.^109 * - 



USER EVALUATION OF INFORHATION SERVICES 
(NATIONAL SCIENCE FOUNDATION CONTRACT C-1027) 



^ This Is a survey coriducted for NSF on "Imprpvements in the Dissemination^ and 

Use of ^ientific and Technical Information", Its aim is (a) to explore the val\^e 
systems which users of information services apply in deciding which information servi/2iif^ 
to buy or use and (b) to provide a guide ^r user selections, among the many compffttidg 
services available. "Service" is def i-ned .herein as any means of access to the^ lifer- ■ , 
ature other than your own reading of journals or books. This may be an abstracting/ 
indexing service, .or the pejrsonal services of a literature searcher/analyst. ^ , 



This questionnaire is addressed to a randpm sample of Exxon professionals at 
Lind en land Florham Park who are known users of information services which th^ Company *, 
provides* Its purpdse is to examihe (he ^b^sis^ upon which you ordlnaril]^^ decide which* 
service(s) you like best, for a specific use. T^^hone enquires about this /orm are 
welcome; your answers can be given by phone, or bjPlnterview if you prefet (ERC j:2555) . 



Yout Division 



Year eAployed by Exxon* 



First cbllege degree and .year 



2. During 1975776, how often have" you a^ked someone to carry out for you any l^iid of 
information search (estimate)? , ' ♦ 



Reques t ed f rom . friames ) 



* ^umber and Complexity o^ Searcheq 1, ' ^ 

for a for a question- for a report lo , 

specified fact; ^ redefined include analysis 

reference, or copy during search ' * of information * 



(Transmitted- suggestions) 



) 



( ) 



^ ^ Please' enter on^ the last line above the approximate number^f these searched 
vher^ you were transmitting a request suggested by someon^ else. |[' ^ 
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Durj 



rtn^ 12.75-76, how often have you 
ll speclTic information service? PJ 
/ vhdch you particularly ^MVe and us< 



US€ 



0 » never 

1 - at^east 



>nce 



yourself or asked someone to *use for you 
add thef names of other services not listed. 



3. •• often (average ijnce a«week) 
4 » dally (constant use)^;^. 



Chemical Abstracts ^ 
. Atl Abstracts 
Derwent Patents. 
Science Citation Inde 
Predicasts 
Engineering Index 



Lockheed/Dialog 
Medlars/Medli-ne/ 
SDC/OrbvLt 



Frequency 
0/12 3 4 



: -4 



-A^^ 



, " . • . , Frequency 

^ ♦ " 0^1 2 3 4 

Aijr Pollution. Abstfacts 

^ 'Applied Sci/TecV Index 

Current Contents* ^ - ^ _ 

^ Environment Abstracts _ j 

Petroleum Abstracts * ; _ 

• ' ERD4 Energy' Abstracts ' 

NpS/GlU Index _ _ 



1, 
2, 



A. .How d^ydfi yite the' sefv-ic^s tt^t you knf)w. in the abqve list (3), agide from' tj)e 
question subject ar6a? ' Please enter your' list b^iow, by name' or number, aod^ 
facjLor for which you have an opi^i^on, on a ^scale from 0 = poor; 1'* 
■^^satifefactory; 3 » good; to ^ « excelleht. ' 



2 



Service* ^ ' 
(by name Qf nUroberV^ 



Chemical Abstjracts 
API ^ 



Quarity Fm:t6rs Budget Factors 



u 
o 



o 

0) 3 



CO 0) 

S. e$ 

CO 

0) o 

1^ IS 



CO 
U 

u 

(0 



c 

0) 

» u 
o ^ 

U 0) 



^ 0) ^,0) 

O CO £ 00 

C 4J ed . 

Q its I-I 

0) p. TO 0)/ 
0) CO 0) > 

a 0) ^ o 

CO 1-1 .CO 



Judgmeot 
" Factors 



C CO 



(0 

u 

>4 a 



Column 
^ (XO)* 




A 



^l^ase enter' in ColUmrPfo (rf) y6ur estimate 6'f , how much this service costs, per year, 
and (b) otHer reasons Why. ypu particularly like or dcj not like this pne*(u8e footnotes 
as ne^ij^y}^ * • 
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5. What tAC t(3ii^ are most important to you in deciding what information service(s) to use? 
- Since*ciiis may Vary with your intended use, pleaSe specify this,*and indicate for 
each type of search Vhich factors you rank as most important in your sel^e-eTqn'">f 
services (on a scale 0*to A, ^ero to vital). Consider all quality factors as a ^(pup, 
all budget factors, and ail jodgme^nt facCoi^, as ^separate dimensions in the selection: 
process:, ' - • , ' * • ^ ' • 



Type of .S^ rch 



Information seajr 
for 



intended 
use* 



Relative importance (0, 1,2,3,4)' of 
factors in selecting- I 
a service for this use: 



Usual % of references 
you want to as 



quality^ budg.et judgmeht * -f title/ 

(convenience) ( costs) (reputation) originals abstracts author , 



a specified fact, 
reference ^ * 



patent information 
state of the art 

m 

review , 
» 

browsing," 

general background 



project selection 



■* *5T>ecl?^v,- as use for R/P (^la-inL-^p ^cu^ilj); rcpor t (writing); client (fo#: sending to soVeone) ; 
uerisicyi (management); other . " . . r * 



6. In your evaluation of information services, hov* impqrtaot'is i.t ift your opinion to 

— ^ - o ' + • 



(prefer not) ' (not needed) , (neutral)* (yes) (vital) 



.know the individual who 
will do tlie searching 

use a service I have' 
used before 

know myself how" to do 
compu ter^earclfing 

know approximate .costs 
'(order of magnitude) 

know eltfc t"*dollar tps ts 
/br comparin'g .s cervices 
* . * 

rank accuracy ahead o€ 

compile teness' ot coverage 
/ ^ <■ 

get k:ey references promptly, 

• bef6re the fjjial report 




V. 



I 
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. Other + or - factors which seem significant -to m^i»lncludej 



What effect does It have on you If a service which gives good results is ' , 

expensive? • * " • V 

•■'•); ' • ' • , 

oif/hardv,to use? ' "» , 

t 

r " ^ . , ► 

of uncertain reputation? * * * 

» 



xoataina hidden errors ^ * . ^ I ' , , . - 



Why do you ^Ink people particularly like to use or avoid pertain types of 
information services? . ' . 



4 



. .Tbis survey'of user, values systems is in no way a reflectioji on the 

inforBtion services supplied by ESSE. Thank you for your time ahd^terest.* 
It will help, if you Qign, buXlf ^yoj^ don' t want tQ, leave ^t off. 



Signe d • > ^ * ( optional) *, 
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